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Abstract 

This  study  investigated  the  types  of  processes  used  by 
certain  U.S.  government  agencies  to  provide  /Wessons 
learned!''  that  could  be  used  by  USAF  research,  development 

and  acquisition  project  managers.  Areas  of  interest  include 

- 

the  types  of  agencies  that  could  provide  "lessons  learned ;"y 
the  sources,  methods,  and  techniques  these  agencies  use  to 
process  (acquire,  maintain,  and  disseminate)  management 
information  such  as  lessons  learned;  and  the  measurement 
criteria  the  agencies  use  to  evaluate  the  effectiveness  of 
these  processes.  A  general  literature  search  helped  to 
identify  eleven  candidate  agencies;  eight  were  chosen  for 
full  investigation.  A  survey  of  each  "sample^'  agency  was 
conducted.  The  survey  obtained  '^agency  description'^ 
information/materials  for  review,  then  used  follow-up 
interviews  to  clarify/update  that  information.  Final 
information  was  analyzed  using  nominal  sorting  and  scoring 
techniques.  Results  indicated  "information  gatherer"  and 
"resource  provider"  agencies  tended  to  have  "lessons 
learned"  programs,  while  "ievaluator'l  and 

"oversight/management'i  agencies  did  not.  A  similar  pattern 
was  seen  regarding  the  use  of  "information/lessons  learned" 
process  "metrics."  Several  changes  to  improve  "lessons 
learned"  processes  were  suggested. 
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AN  INVESTIGATION  OF  CERTAIN  U.S.  GOVERNMENT 


'•LESSONS  LEARNED"  PROCESSES  AND  THEIR 
APPLICATION  TO  USAF  RESEARCH, 
DEVELOPMENT,  AND  ACQUISITION  PROJECT  MANAGERS 


I.  Introduction 


General  Issue 

As  a  result  of  reduced  tensions  between  the  United 
States  (U.S.)  and  the  Soviet  Union,  the  U.S.  Air  Force 
(USAF)  is  faced  with  shrinking  budgets  in  the  1990s.  As  the 
resources  provided  by  those  budgets  also  shrink,  the  USAF  is 
reorganizing  its  existing  research,  development,  and 
acquisition  (RD&A)  structures.  For  example,  July  1,  1992  is 
the  officially  scheduled  activation  date  for  the  Air  Force 
Materiel  Command  (AFMC) .  This  new  agency  will  be 
responsible  for  the  RD&A  functions  currently  performed  by 
Air  Force  Systems  Command  (AFSC)  and  the  logistics  support 
functions  currently  accomplished  by  Air  Force  Logistics 
Command  (AFLC) .  The  AFMC  "will  combine  AFSC's  expertise  in 
science,  technology,  research,  development,  and  testing  with 
AFLC's  expertise  in  life-cycle  acquisition  and 
supportability  to  create  one  organization  for  weapon  systems 
management"  (1:76).  The  current  RD&A  organizati-- ns  and 
personnel  shall  have  to  learn  how  to  perform  their  functions 


more  effectively  and  efficiently,  or  face  the  discontinuance 
of  their  operations.  The  skills  and  knowledge  of  the  USAF 
RD&A  personnel  shall  be  key  factors  in  these  reorganization 
efforts. 

In  a  19  March  1990  letter  to  the  Assistant  Secretary  of 
the  Air  Force  for  Acquisition,  General  Bernard  Randolph 
(then  Commander  of  AFSC)  stated  "...we  know  that  improved 
training  at  all  levels  is  going  to  be  essential  as  we  enter 
an  environment  of  reduced  resources”  (2) .  Personnel  need  to 
understand  the  responsibilities  of  their  duties,  to  improve 
the  productivity  and  quality  of  their  efforts. 

In  October  1989,  Gen.  Randolph  had  established  the 
Acquisition  Process  Excellence  (APEX)  Team.  The  APEX  Team 
was  to  examine  the  existing  acquisition  system,  identify 
unnecessary  subsystems  or  procedures  that  could  be  improved, 
and  recommend  appropriate  changes.  Two  "improvable"  areas 
identified  by  the  APEX  Team  were  training  and  streamlining 
(3) .  In  the  training  area,  all  types  of  training  are  to  be 
reviewed  and  revised  (as  needed)  to  effectively  support 
providing  the  required  knowledge  and  skills  to  RD&A 
personnel.  In  the  streamlining  area,  the  existing  Total 
Quality  Management  (TQM)  approach  will  provide  for,  among 
other  things,  "unity  of  action  (do  it  once,  report  it  once)" 
in  various  tasks  such  as  "consolidating  the  lessons  learned 
process"  (3) .  Therefore,  the  existing  "lessons  learned" 
procedures  and  processes  should  be  reviewed  and  revised,  to 
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reduce  duplication  of  efforts  and  to  improve  the  training 
and  professional  development  of  RD&A  personnel. 

Problem  Statement 

There  is  no  coordination  between  those  organizations 
inside  (and  outside)  the  Air  Force,  that  possess  "lessons 
learned"  information,  regarding  the  most  effective  methods 
to  process  this  information  such  that  RD&A  project  managers 
can  use  it  to  enhance  their  skills  and  capabilities.  The 
purpose  of  this  study  is  to  identify  possible  improvements 
to  the  current  "lessons  learned"  processes.  Such 
improvements  would  enable  RD&A  project  managers 
(particularly  those  that  are  new  to  the  RD&A  career  fields) 
to  more  effectively  gain  knowledge  from  the  "lessons 
learned"  by  others  in  those  fields. 

Research  O^:.^stions 

1.  What  agencies  do  (or  can)  obtain  information  that 
is  (or  can  be)  used  in  "lessons  learned"  that  are  pertinent 
to  the  responsibilities  of  RD&A  project  managers? 

2.  What  sources,  methods,  and  techniques  do  these 
agencies  use  to  "process"  (acquire,  maintain,  and 
disseminate)  this  information? 

3.  What  criteria  do  these  agencies  use  to  judge  the 
effectiveness  and  efficiency  of  their  information  processing 
efforts  (particularly  with  respect  to  "lessons  learned"  and 
the  needs/desires  of  information  "customers")? 
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4.  Based  upon  these  criteria  (and  other  criteria,  such 
as  suggested  by  customers  perhaps) ,  what  possible  procedures 
and  techniques  could  be  used  to  improve  these  agencies' 
current  methods,  such  that  the  support  of  RD&A  project 
managers  would  be  enhanced? 

Scope 

Agencies  to  be  investigated  during  this  study  shall  be 
limited  to  those  whose  missions  allow  them  to  manage/direct, 
or  provide  resources/support,  or  evaluate/advise,  or 
consolidate  project  information  obtained  from,  USAF  RD&A 
organizations  or  programs.  Therefore,  those  numerous  RD&A 
agencies  at  or  below  System  Program  Office  or  equivalent 
levels  (including  education  and  training  agencies  such  as 
the  Air  Force  Institute  of  Technology,  the  Defense  Systems 
Management  College,  the  Systems  Acquisition  School  at  Brooks 
AFB  TX,  and  others)  which  would  usually  create  or  use 
(rather  than  process)  "lessons  learned"  information,  will 
generally  be  excluded  from  direct  investigation  (unless 
"creator/user"  data  is  required) .  Rather,  those  types  of 
agencies  shall  be  considered  as  the  "customers"  (as  well  as 
sources)  for  such  information.  Also  excluded  from  direct 
investigation  are  most  non-USAF  agencies,  except  those  with 
missions  as  described  above. 

The  specific  agencies  to  be  investigated  will  be 
identified  in  Chapter  III,  Methodology. 


4 


Definitions 

A  list  of  definitions  of  certain  terms  used  throughout 
this  study  is  provided  below.  In  those  instances  where 
other  agencies  define  these  terms  differently,  their 
specific  definitions  will  be  provided  in  the  context  of  the 
relevant  subject  matter. 

Lessons  Learned:  Knowledge  acquired  by  individuals  or 
groups,  regarding  positive  or  negative  aspects  of  efforts 
performed  by  themselves  or  others.  A  simple  example  of  a 
"positive  lesson  learned"  would  be  "Use  of  colorful  charts 
can  often  improve  the  clarity  of  a  project  briefing."  A 
simple  example  of  a  "negative  lesson  learned"  would  be 
"Mislabelled  charts  can  often  cause  delays  or  disruptions  of 
a  project  briefing." 

RD&A  Project  Managers:  Any  individuals  in  an 
organization  who  are  responsible  for  managing  all  (or  most) 
of  the  RD&A  activities  relating  to  (a)  specific  system{s) , 
subsystem (s) ,  or  component (s)  intended  for  one  or  more 
programs/projects.  Examples  would  include:  the  director  of 
a  "regular"  (single  program)  or  a  "basket"  (multiple 
program)  System(s)  Program  Office  (SPO) ;  the  chief  of  the 
project  management  directorate  in  a  SPO;  a  subsystems 
development  manager  in  a  SPO's  project  management 
directorate;  a  project  engineer  acting  as  the  program 
manager  for  a  technology  development  program  in  a 
Laboratory;  a  logistics  specialist  acting  as  the  project 
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manager  for  systems  support  equipment  in  a  SPO  (or  an  Air 
Logistics  Center) ;  and  many  others. 

Electronic  Database  System:  an  information  storage/ 
management  system,  normally  hosted  on  a  large  capacity 
"mainframe"  computer,  where  access  to  the  system  is  achieved 
through  electronic  links  (telephone  signals,  direct  electric 
wiring,  and  so  forth)  with  a  "remote"  computer  terminal. 

Summary 

This  chapter  has  discussed:  the  general  issue  of  the 
need  for  improvements  in  the  RD&A  organizational 
effectiveness;  the  problem  of  lack  of  coordination  for  the 
"lessons  learned"  processes,  and  the  purpose  of  this  study, 
to  identify  possible  improvements  to  those  processes;  and 
the  research  questions  that,  -when  answered,  will  suggest 
such  improvements . 

Chapter  II  shall  discuss  a  general  review  of  the 
literature  pertaining  to  the  USAF's  "lessons  learned" 
processes.  Chapter  III  shall  describe  the  methodology  for 
obtaining  the  data  needed  to  answer  the  research  questions, 
including  an  outline  of  the  data-gathering  methods  and  a 
definition  of  "sample"  organizations.  Chapter  IV  shall 
provide  a  detailed  description  of  the  findings  gathered  from 
the  samples,  and  an  analysis  of  the  findings  with  respect  to 
the  original  problem.  Chapter  V  will  discuss  conclusions 
that  can  be  drawn  from  the  findings  and  analysis,  and 
provide  recommendations  for  subsequent  actions. 
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II .Literature  Review 

Introduction 

This  literature  review  shall  discuss  the  general 
literature  discovered  that  addresses  USAF  "lessons  learned" 
processes,  particularly  those  documents  that  identify  the 
organizations  responsible  for  "lessons  learned"  and  the 
procedures  and  techniques  they  use  to  obtain,  format,  and 
distribute  "lessons  learned"  information.  It  shall  also 
address  the  limitations  encountered  in  the  literature 
review,  and  conclusions  about  the  methods  needed  to  gather 
data  for  this  study. 

Literature  Sources 

Information  sources  initially  utilized  for  this  general 
literature  review  included  the  Air  For'ce  Institute  of 
Technology  (AFIT)  Library,  the  Wright  State  University 
Library,  the  AFIT  Systems  Acquisition  Department  (AFIT/LSY) , 
and  the  Advanced  Tactical  Fighter  System  Program  Office. 

General  Literature 

The  Defense  Technical  Information  Center  (DTIC)  is  one 
identified  source  for  certain  types  of  "lessons  learned" 
information.  This  organization  accepts  most  technical 
reports  generated  by  Department  of  Defense  (DOD)  agencies, 
including  those  reports  that  contain  "lessons  learned."  An 
individual  can,  through  his  local  DOD-af filiated  technical 
library  (this  researcher  used  the  AFIT  Library) ,  request  a 
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Technical  Report  Summaries  listing  from  DTIC.  The  form  used 
to  make  this  request  requires  several  key  "identifier”  or 
"descriptor"  phrases  or  words  be  provided,  for  use  by  the 
computerized  search  program  software.  This  software  looks 
for  the  same  key  words  or  phrases  in  the  "identifier"  and 
"descriptor"  blocks  utilized  in  the  beginning  of  all  DOD 
technical  reports.  If  "lessons  learned"  is  one  of  a 
researcher's  key  phrases,  and  is  also  associated  with  a 
specific  report,  that  report  will  be  identified  in  the 
Technical  Report  Summaries  Listing;  the  title,  an  abstract, 
and  the  descriptor  and  identifier  words  are  shown.  If  the 
researcher  also  uses  key  phrases  such  as  "acquisition 
management,"  "research  and  development  management,"  and 
"system's  management,"  plus  other  identifiers  (such  as 
"contracts  management,"  for  example),  then  the  DTIC  search 
program  can  also  find  reports  that  utilize  those  phrases  as 
descriptors  or  identifiers.  Once  the  DTIC  report  number  and 
title  is  known,  the  desired  reports  can  be  ordered  from  DTIC 
through  the  technical  library. 

The  major  difficulty  with  using  the  DTIC  system  as  a 
source  for  RD&A  project  management  "lessons  learned"  appears 
to  be  the  perception  of  DTIC  by  the  users.  DTIC  is 
generally  regarded  as  a  reliable  source  for  specific 
technical  information  (such  as  reports  explaining  various 
technical  achievements  on  various  programs)  rather  than 
"non-technical"  managerial  information.  As  such, 
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management-specific  information  is  apparently  not  frequently 
submitted  to  DTIC. 

This  conclusion  arises  from  an  examination  of  this 
researcher's  Technical  Report  Summaries  Listing  (which  used 
all  previously  mentioned  descriptors  and  identifiers) ;  only 
13  reports  contained  any  of  these  descriptors  other  than 
"lessons  learned." 

The  Acquisition  Logistics  Division  -  an  organization 
within  the  Air  Force  Logistics  Command  -  at  Wright-Patterson 
Air  Force  Base  in  Ohio,  contains  an  office  responsible  for 
"lessons  learned"  information.  A  pamphlet  published  by 
ALD/LSL  identifies  that  office  as  "the  Air  Force  focal  point 
for  Lessons  Learned"  (4) .  The  pamphlet  provides  Autovon 
telephone  numbers  for  verbal  or  computer  contact,  and 
describes  procedures  and  formats  for  providing  "Lessons 
Learned"  to  that  office,  for  inclusion  into  their  database. 
The  pamphlet  highly  encourages  readers  to  furnish  "lessons 
learned"  to  that  office,  and  to  obtain  "lessons  learned" 
information  from  the  office,  as  well. 

Other  agencies  responsible  for  "lessons  learned" 
information  include  the  General  Accounting  Office  (GAO) ,  and 
the  AFSC  and  USAF  Inspector  General  (IG)  offices.  As 
profiled  in  "Project  Management"  (SMGT  646) ,  a  graduate 
level  course  taught  in  the  AFIT  School  of  Systems  and 
Logistics,  one  of  the  many  functions  performed  by  "auditing 
and  evaluation"  agencies,  such  as  the  GAO,  the  Air  Force 
Audit  Agency  (AFAA) ,  and  the  AFSC  and  USAF  IGs,  is  to 
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acquire  "lessons  learned"  information  from  the  various 
programs  or  organizations  they  visit  (5) .  Much  of  this 
information  concerns  RD&A  program  management  activities. 

For  example,  the  Air  Force  Inspection  and  Safety  Center, 
Directorate  of  Inspection  (AFISC/IG) ,  at  Norton  Air  Force 
Base  in  California,  "assists  the  Inspector  General  of  the 
Air  Force"  to  "examine  and  analyze"  and  to  "report  on"  the 
Air  Force's  "operational  readiness  ...  and  systems 
management  effectiveness"  (6:100).  Also,  the  AFAA's 
Acquisition  and  Logistics  Audit  Directorate  (AFAA/QL) ,  at 
Wright-Patterson  Air  Force  Base,  Ohio,  "directs  the 
development  and  management  of  multisite  audits  related  to 
acquisition,  weapon  systems,  ..."  (7:108). 

Acquisition  program  "oversight"  agencies,  such  as  the 
Air  Force  Program  Executive  Offices  (AFPEOs) ,  also  have 
access  to  information  suitable  for  project  management 
"lessons  learned."  This  access  is  due  to  the  fact  that  the 
six  Program  Executive  Officers  (PEOs)  act  as  the  "general 
managers  for  clusters  of  major  programs"  and  therefore  can 
obtain  (and  review)  detailed  programmatic  information  from 
their  various  programs'  managers  (8:13). 

Headquarters  organizations  such  as  HQ  Air  Force  Systems 
Command  -  at  Andrews  Air  Force  Base,  Maryland  -  are  also 
regarded  as  agencies  responsible  for  "lessons  learned."  The 
Acquisition  Process  Excellence  (APEX)  study  briefing  charts 
addressed  the  important  role  that  HQ  AFSC  would  have  to  play 
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in  providing  knowledge  and  resources  to  support  the  change 
to  a  smaller,  more  efficient  acquisition  system  (3) . 

In  addition,  there  are  several  agencies  within  the 
various  "product-oriented”  Systems  Divisions  in  AFSC,  that 
have  the  capabilities  to  obtain  "lessons  learned" 
information.  For  instance,  three  such  organizations  in  the 
Aeronautical  Systems  Division  (ASD)  at  Wright-Patterson  Air 
Force  Base  in  Ohio  are:  the  Total  Quality  Management  office 
(ASD/TQ) ,  the  History  Office  (ASD/HO) ,  and  the  Deputy  Chief 
of  Staff  (DCS)  for  Program  Management  (ASD/CY) .  The  HO  and 
TQ  offices  can  obtain  various  program-related  information 
from  the  SPOs  and  other  agencies,  to  use  as  part  of 
historical  records  (HO)  or  quality  improvement  efforts  (TQ) , 
The  CY  office,  which  is  structured  similarly  to  the  other 
"functional  support"  offices  (such  as  the  Comptroller, 
Contracting,  and  the  Integrated  Engineering  &  Technical 
Management  DCS  offices) ,  would  be  expected  to  obtain  the 
management  information  needed  to  provide  well-trained, 
knowledgeable  project  management  personnel  to  the  ASD  SPOs 
(9)  . 

An  R&D  oversight  agency  at  ASD,  Wright  Laboratories, 
would  also  be  expected  to  have  access  to  information  for 
"lessons  learned."  The  Wright  Laboratories  office  is  the 
management  focal  point  for  the  efforts  of  six  formerly 
separate  Air  Force  laboratories;  it  is  expected  to 
coordinate  the  development  of  the  various  flight  dynamics, 
materials,  avionics,  aero-propulsion,  electronic,  and 
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armament  technologies  which  in  turn  support  the  development 
of  aircraft/munition  weapon  systems  (10). 

Limitations 

While  the  aforementioned  references  are  very  useful  in 
focusing  this  study,  it  is  readily  apparent  that  this 
literature  review  did  not  account  for  all  the  documentation 
(that  is  assuredly  available)  that  would  address  information 
for  "lessons  learned"  (for  project  managers) ,  directly  or 
indirectly.  In  fact,  most  of  the  references  cited  would  not 
be  obtainable  using  a  "standard"  literature  search  as 
described  in  Emory's  "Research  Methods"  text  or  in  AFIT 
Library's  instructional  pamphlet  concerning  searches  for 
data  &  information. 

Conclusions 

Additional  documentation  describing  the  information  and 
"lessons  learned"  processes  used  in  several  of  the 
organizations  cited  should  be  available  from  those 
particular  organizations.  Such  documentation  could  include 
mission  statements,  operating  instructions,  policy  letters, 
newcomer  introduction  presentations,  regulations,  program 
review  reports,  project  briefings,  and  several  other  types 
of  documents  not  normally  available  from  a  "standard" 
literature  search.  It  would  therefore  be  expected  that 
additional  documents  would  be  obtained,  once  these 
organizations  were  contacted  and  such  documentation  became 
accessible.  Information  obtained  from  documents  provided  by 
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the  contacted  organizations  shall  be  discussed  in  Chapter 
IV,  Findings  and  Analysis. 

Summary 

This  chapter  described  the  results  of  a  general 
literature  review,  limitations  of  that  review,  and 
conclusions  drawn  based  on  that  review.  Chapter  III  will 
discuss  the  methodology  for  obtaining  needed  data. 
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III.  Methodology 

Introduction 

This  chapter  shall  discuss  the  general  methods  of 
survey  and  observation  that  were  used  to  gather  the  data 
needed  for  this  study.  Justification  for  the  use  of  these 
methods  shall  involve  describing  the  disadvantages  of  purely 
observational  methods  and  the  advantages  of  the  "survey  plus 
observation"  methods.  The  population  and  samples  of 
interest  shall  be  defined  in  terms  of  universe,  population, 
sampling  subdivisions  and  supporting  assumptions.  The 
survey  steps  -  initial  contact,  information  gathering,  and 
"follow-up"  interviews  -  and  the  survey  instrument  -  an 
interview  "topic  checklist"  -  shall  be  discussed,  in  terms 
of  purpose,  sequence  and  reliat^lity.  A  "topic  checklist" 
format,  that  discusses  the  basis  for  deriving  the  interview 
questions,  shall  be  presented  (the  actual  topic  checklist, 
along  with  an  "initial  contact  guidelines"  sheet,  will  be  in 
appendices) .  An  administrative  plan,  which  will  describe 
data  gathering  and  grouping  procedures,  shall  be  outlined. 

General  Methods  To  Be  Used: 

Data  needed  for  this  study  were  gathered  using  a 
combination  of  survey  and  observational  methods.  The 
specific  survey  techniques  used  were  personal  interviews  and 
telephone  interviews.  Observational  methods  were  employed 
before  and  during  the  survey  process. 
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Justification  of  Chosen  Methods; 

The  detailed  aspects  of  the  USAF's  various  "lessons 
learned"  efforts  are  not  specifically  described  in  the 
journals,  reports,  and  other  documents  available  from  a 
general  literature  search.  Such  descriptive  data  must  be 
gathered,  primarily  from  the  agencies  that  are  responsible 
for  "lessons  learned"  processes.  Interviewing  individuals 
within  these  agencies  offers  opportunities  to  locate  and 
obtain  documents  and  other  information  that  can  be  reviewed 
using  observational  methods.  In  addition,  interviewing 
allows  the  researcher  to  obtain  detailed  information,  give 
(or  receive)  clarification  regarding  specific  details,  and 
pursue  areas  of  interest  newly  identified  during  the 
document  reviews  or  interviews. 

Population/Samole 

The  universe  of  interest  consists  of  those 
organizations  that  can  provide  "lessons  learned"  information 
that  could  be  useful  to  Air  Force  RD&A  project  managers. 

This  would  include  any  agencies  that  support  the  development 
of  technologies  and  acquisition  of  systems  that  the 
Department  of  Defense  (DOD) ,  and  the  USAF  in  particular, 
needs  to  perform  its  missions. 

The  population  of  interest  consists  of  those  agencies 
(from  the  universe  of  interest)  that  oversee/manage  many 
RD&A  projects/organizations,  provide  resources/support  to 
RD&A  programs,  evaluate  the  progress/status  of  RD&A 
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projects,  or  gather/maintain  information  about  RD&A 
efforts.  The  samples  of  interest  shall  be  taken  from  each 
of  four  subdivisions  of  the  population  of  interest.  Those 
subdivisions  are:  agencies  which  have  oversight/management 
responsibility  for  RD&A  programs;  agencies  which  have 
"support  provider"  responsibility  for  RD&A  programs,  but 
which  do  not  have  oversight  responsibility;  agencies  which 
have  evaluation  responsibility  for  RD&A  programs,  but  which 
do  not  have  oversight  or  support  responsibilities;  agencies 
which  have  responsibilities  to  acquire  RD&A  program 
information,  but  which  do  not  have  oversight,  support,  or 
evaluation  responsibilities. 

The  selection  of  the  population/sample  characteristics 
is  based  on  the  following  assumptions: 

1.  The  purpose  of  RD&A  project  managers  is  (primarily) 
to  plan,  organize,  coordinate,  direct,  and  control  the 
activities  in  particular  projects  such  that  the  desired  end 
product  is  successfully  obtained; 

2.  "Oversight"  agencies  require  accurate,  detailed 
information  concerning  the  status  of  the  project,  especially 
specifics  about  necessary  actions  regarding  management  of 
costs,  schedules,  performances  and  reliability  & 
maintainability  of  the  items  under  development; 

3.  "Provider"  agencies  also  require  accurate  and 
detailed  information,  particularly  specifics  about  necessary 
actions  regarding  the  furnishing  of  facilities,  resources 
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and  personnel  (to  support/accomplish  the  required  program 
tasks/mission) ; 

4.  "Evaluator"  agencies  need  similar  (but  usually 
somewhat  "broader")  information,  in  order  to  judge  the 
effectiveness  of  the  project  management  actions  and  the 
project  in  general; 

5.  "Information-gathering"  agencies  have  the 
capability  to- collect  "management  actions"  data,  in  addition 
to  general  program  information  or  specific  technical  details 
about  the  product  development. 

At  least  two  samples  have  been  taken  from  each  of  the 
four  population  subdivisions.  The  agencies  chosen  as 
initial  samples,  by  subdivision  category,  were:  the  Air 
Force  Program  Executive  Office  for  Tactical/Airlift  - 
AFPEO/TA  -  and  the  AFPEO  for  Tactical  Strike  -  AFPEO/TS  -  as 
"overseers;"  Headquarters  AFSC  and  Aeronautical  Systems 
Division's  DCS  for  Program  Management  office  -  as 
"providers;"  General  Accounting  Office  and  Air  Force 
Inspection  and  Safety  Center's  Directorate  for  Inspection  - 
as  "evaluators;"  Acquisition  Logistics  Division's  "Lessons 
Learned"  office  and  ASD's  Total  Quality  Management  office  - 
as  "gatherers."  These  agencies  were  assigned  to  the 
specific  subdivisions  based  on  the  similarities  between 
their  generally  understood  purposes  and  the  definitions  of 
the  population  subdivisions.  These  particular  agencies  were 
selected  as  samples  because  each  agency  has  the 
responsibility  (as  required  by  regulation  or  mission  needs) 
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to  interface  with  substantial  numbers  of  RD&A  organizations 
inside  (and  outside)  their  parent  command/ agency  and/or  the 
USAF  in  general.  For  example,  the  AFPEO/TA  office  oversees 
the  C-17  advanced  airlifter  program  and  the  Advanced 
Tactical  Fighter  (ATF)  program,  both  of  which  have  ether 
service  users  (Army  for  C-17,  Navy  for  ATF)  in  addition  to 
the  USAF;  the  AFPEO/TS  office  oversees  the  Advanced  Medium 
Range  Air  to  Air  Missile  (AMRAAM)  and  the  Joint  Surveillance 
and  Target  Attack  Radar  System  (JSTARS)  programs,  which  will 
benefit  the  USAF  and  Navy  (AMRAAM)  and  the  Army  (JSTARS) 
(11:119-120) . 

Other  agencies  that  have  been  included  (to  some  degree) 
in  the  investigation  (in  order  to  act  as  additional  samples 
or  "replacements"  for  initial  samples,  if  needed)  were  the 
AFAA/QL  and  the  AFSC  IG  (as  evaluators) ,  the  ASD/HO  (as  a 
gatherer) ,  and  the  Wright  Laboratories  management  office  (as 
an  overseer) .  These  "reserve  samples"  were  needed  in  case 
insufficient  (or  unusable)  information  was  obtained  from  a 
"primary"  sample  agency,  or  to  provide  for  those  instances 
where  an  initial  sample  agency  might  be  composed  of  smaller 
agencies  (some  of  which  would  more  accurately  reflect  the 
sampling  subdivisions)  such  that  the  smaller  organizations 
should  be  used  as  the  samples.  Such  a  situation  would 
possibly  call  for  other  samples  to  be  added,  to  maintain  a 
rough  symmetry  in  the  number  of  samples  taken  from  each 
sampling  subdivision.  The  investigation  of  "reserve"  sample 
agencies  was  performed  using  only  survey  steps  1  and  2;  step 
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3  would  be  taken  only  if  "new”  samples  were  required.  These 
steps  are  described  in  the  following  subsection. 

Survey  Steps 

The  first  step  in  conducting  the  survey  (of  each  sample 
agency)  is  to  "make  initial  contact.”  This  contact  is 
intended  to  locate  (and  gain  the  cooperation  of)  the 
office (s)  and  individual (s)  within  that  agency  most  likely 
to  possess  the  desired  information.  The  actions  involved  in 
"initial  contact"  are  introduction  and  explanation. 

Introduction  involves  identifying  the  researcher  and 
describing  the  purpose  of  the  contact.  Providing  the  name, 
organization,  and  "job  description"  of  the  researcher 
establishes  that  this  person  is  also  a  "member"  of  the  U.S. 
Government,  and  therefore  may  be  allowed  access  to  U.’S. 
Government  offices,  workers,  a'nd  information. 

Explaining  the  thesis  research  in  clear  terms  -  "what 
information  do  I  want,  why  do  I  want  it,  and  'who'  am  I 
trying  to  get  that  information  from"  -  enables  the  contacted 
individuals  to  determine  whether  or  not  they  (or  someone 
else  in  their  offices)  are  able  to  assist  the  researcher. 

If  they  cannot  help,  those  "contactees"  are  often  able  to 
refer  the  researcher  to  other  individuals/offices  that 
"might  be  of  more  help." 

Once  the  researcher  has  located  the  "suitable" 
offices/individuals,  then  a  "survey  procedures  explanation" 
is  required.  The  researcher  (re) explains  the  purpose  for 
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the  survey  (if  needed),  then  describes  the  intended  methods 
for  obtaining  the  desired  information.  These  descriptions 
are  summaries  of  the  survey  steps  2  and  3,  which  are 
discussed  in  the  following  paragraph. 

The  second  step  in  surveying  each  sample  agency  is  to 
"begin  gathering  the  information.”  In  this  step,  the 
researcher: 

(a)  requests  detailed  information  about  the  agency  and 
its  operations  -  "what  does  your  organization  do,  and  how 
does  it  do  it? ; ” 

(b)  (re) specif ies  the  primary  areas  of  interest  - 
information  processing,  lessons  learned,  support  for  RD&A 
project  managers; 

(c)  provides  examples. of  the  types  of  sources  that 
could  contain  such  information  -  agency  briefings,  policy 
letters,  program  review  presentations,  "newcomer 
orientation"  charts/ reports ,  operating  instructions,  and 
other  similar  sources; 

(d)  emphasizes  the  desirability  of  this  information 
being  provided  to  the  researcher  in  document  form  (if 
feasible)  -  documents  help  minimize  the  individuals'  "time 
away  from  their  regular  job,"  and  allow  the  researcher  to 
fully  review  (and  re-review  as  needed)  areas  of  interest, 
and  also  would  aid  the  researcher  in  focusing  the  next  step 
in  the  survey,  the  "follow-up"  interviews; 

(e)  explains  that  information  not  contained  in  these 
sources  (or  source  information  that  is  not  "clearly" 
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understood  by  the  researcher)  will  be  the  major  concern (s) 
of  the  "follow-up"  interviews; 

(f)  makes  arrangements  to  obtain  and  review  the 
information  (and  to  identify  the  "interviewees"  for  the 
"follow-up"  sessions) . 

The  third  step  involved  in  conducting  each  survey  is 
done  after  the  "initial  source"  information  has  been 
reviewed.  This  third  step  is  "conduct  the  follow-up 
interviews."  In  this  step  the  researcher: 

(a)  contacts  the  individual  interviewee (s)  and 
arranges  interview  sessions; 

(b)  starts  the  interview  by  (re) explaining  the  reasons 
for  the  interview (s)  -  to  confirm/modify  previously  obtained 
information,  to  clarify  such  information  as  was  "unclear," 
and  to  obtain  any  information  that  was  previously 
unavailable; 

(c)  states  the  interview  questions  (which  were  derived 
from  the  "topic  checklist") .  The  topic  checklist  is 
described  in  detail  in  the  following  section.  Checklist 
Survey  Instrument. 

(d)  if  the  researcher  has  data  that  appears  to  answer 
the  question (s),  that  data  is  described  to  the  interviewee; 
that  individual  is  requested  to  confirm/correct  that  data; 

(e)  if  the  researcher's  "question  answering"  data  is 
"unclear,"  the  researcher  explains  the  data  and  the  reasons 
why  the  data  appear  unclear,  then  requests  clarification; 
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(f)  if  the  researcher  has  no  "answer"  data  for  the 
question (s) ,  the  researcher  states  that  "the  previous 
information  did  not  appear  to  have  addressed  this  issue." 

Once  the  interviewee  has  answered  the  questions,  the 
researcher  briefly  reviews  the  answers  with  the  interviewee, 
to  insure  the  interviewee  was  properly  understood.  The 
researcher  also  arranges  to  contact  the  interviewee  at  a 
later  date,  in  case  more  questions  or  issues  arise. 

Checklist  Survey  Instrument 

The  topic  checklist  is  based  on  the  requirement  to 
answer  the  research  questions,  and  reflects  the  sample 
subdivisions  in  the  population  of  interest.  The  sequence  of 
topics  are  structured  such  that  topics  relating  to  Research 
Question  1  are  addressed  first,  followed  by  topics  relating 
to  Research  Question  2,  and  so  forth  until  all  Research 
Question  categories  have  been  dealt  with. 

Each  person  contacted  in  the  "follow-up"  interviews  is 
asked  questions  derived  from  the  same  topic  checklist.  This 
condition  is  intended  to  promote  the  reliability  of  the 
topic  checklist  as  a  survey  instrument;  use  of  the  same 
checklist  should  eliminate  the  bias  that  could  be  produced 
by  interviewer  alteration  of  discussion  topics  (and 
subsequent  gaps  or  changes  in  answers  from  different 
respondents) .  Although  the  exact  wording  of  the  interview 
questions  can  be  tailored  to  reflect  the  nature  of  the 
specific  agency/ individual ,  the  questions  are  required  to 
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make  clear  the  nature  of  the  topic  under  discussion  and  the 
types  of  information  desired  (through  use  of  examples, 
scenarios,  or  other  descriptive  mechanisms) . 

The  basic  format  of  the  topic  checklist  is  shown  as 
follows; 

Topic  #X:  Topic  Subject  (generally  a  derivative  of  a 
key  phrase  or  subject  in  the  relevant  Research 
Question) . 

Purpose  of  Interview  Questions;  Describes  in  general 
terms  the  information  sought  by  the  questions,  and  what 
overall  "picture"  should  be  provided  by  the  answers. 

Objective  Y;  Specific  types  of  topic-oriented 
information  "goals"  that  the  researcher  must  address. 

-  Prompting  Phrases/Examples;  These  words  act  as 

clarifying  reminders  of  the  additional  level  of 
detail  needed  (in  the  answers)  to  properly 
satisfy  the  objectives. 

The  actual  topic  checklist  is  shown  in  Appendix  A. 

Data  Collection  Plan 

Personal  contact  procedures  were  used  to  gather  data 
from  all  sample  agencies  located  in  the  Wright  Patterson 
AFB,  OH  area.  If  the  researcher's  travel  authorization  was 
obtained,  data  from  other  organizations  was  also  gathered 
using  personal  contact  methods.  If  travel  authorizations 
were  unavailable,  those  organizations  were  contacted  by 
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telephone,  and  data  gathered  using  verbal  communication 
(pertinent  documents  were  gathered  by  mail;  "telefaxes”  or 
couriers  were  options  to  be  used  if  needed) .  This  plan  did 
not  preclude  data  gathering  from  prospective  sources  who  had 
already  traveled  to  the  Wright-Patterson  AFB  area  (for 
meetings,  assignment  changes,  or  other  purposes) . 

Data  from  these  sources  was  to  be  nominally  sorted,  on 
the  basis  of  common  features  as  identified  (using  the 
research  questions  to  focus  the  categories  of  interest) . 

The  data  was  to  be  scored  based  upon  the  nominal  breakout, 
and  a  consolidated  criteria  list  derived  from  each  agency's 
performance  evaluation  criteria  (as  those  criteria  pertain 
to  the  "lessons  learned"  efforts  perfoinned  by  those 
agencies) . 

This  data  collection  plan  depended  on  a  key  assumption: 
that  the  surveyed  agencies  do  indeed  have  some  form  of 
performance  measurement  criteria  that  can  be  readily  used 
(and  are  reliable  and  valid)  to  evaluate  "lessons  learned" 
performance.  If  not,  the  "lacking  agency"  could  have  been 
excluded  from  analysis,  or  that  agency's  data  would  have  to 
have  been  analyzed  using  measurement  criteria  derived  from 
other  agencies'  criteria,  or  some  sort  of  "draft/initial" 
performance  measurement  criteria  would  have  to  have  been 
constructed,  as  a  product  of  the  investigation.  Chapters  IV 
and  V  discuss  the  actions  that  were  required. 
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Summary 


This  chapter  has  identified  the  general  methods  to  be 
used  to  gather  data  (survey,  interviews  and  observation) . 

It  has  discussed  the  justification  for  these  methods  (lack 
of  observable  documents) .  It  has  defined  the  population  and 
samples  of  interest,  explained  how  the  sample  agencies  were 
selected,  described  the  three  steps  involved  in  conducting 
the  survey,  and  discussed  the  "topic  checklist"  survey 
instrument  (the  basis  for  interview  questions  concerning 
specific  research-question-related  categories) .  It  has  also 
outlined  the  data  collection  plan,  explaining  how  the  data 
will  be  gathered  and  grouped.  Chapter  IV  will  deal  with  the 
detailed  survey  findings,  and  analysis  of  the  findings. 
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This  chapter  shall  describe  the  detailed  findings  about 
the  sample  agencies  (as  derived  from  the  literature  review 
and  agency  investigations) ,  and  shall  discuss  the  analysis 
of  their  characteristics.  The  findings  shall  be  presented 
by  sampling  subdivision,  and  shall  describe  each  agency's 
various  attributes  as  they  relate  to  the  topic  checklist  and 
the  research  questions.  Then  the  analysis  of  these  agencies 
shall  be  discussed  in  terms  of  the  nominal  characteristics 
identified,  the  values  assigned  to  those  characteristics  (as 
based  upon  the  research  questions) ,  and  the  "scores"  for  the 
agencies  as  determined  from  those  values. 

Conclusions  drawn  from  the  analysis,  suggested  actions 
based  on  those  conclusions,  and  issues  requiring  future 
research  shall  be  discussed  in  Chapter  5,  Conclusions  and 
Recommendations . 

Findings 

The  findings  by  sampling  subdivisions,  are  presented  in 
the  following  order: 


(a) 

"Gatherer" 

agencies 

(b) 

"Provider" 

agencies 

(c) 

"Evaluator" 

agencies 

(d) 

"Oversight" 

agencies 
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within  each  subdivision,  each  agency's  characteristics 
shall  be  described  in  detail,  before  moving  on  to  the  next 
set  of  agency  (or  subdivision)  descriptions.  These 
arrangements  are  intended  to  provide  the  reader  with  an 
appreciation  of  the  differences  and  similarities  between  the 
agencies  within  the  same  sampling  subdivision.  The 
arrangements  should  also  allow  for  clearer  understanding  of 
the  "cross-sample-subdivision"  relationships  between  certain 
of  the  profiled  agencies. 

Gatherer  Agencies;  The  "gatherer"  agencies  described 
are:  the  Acquisition  Logistics  Division's  Directorate  for 
Support  Equipment  and  Lessons  Learned  (ALD/LSE) ,  and  the 
Aeronautical  Systems  Division's  Total  Quality  Management 
Office  (ASD/TQ) . 

ALD/LSE:  This  agency  is  the  Office  of  Primary 
Responsibility  (OPR)  for  the  Air  Force  Lessons  Learned 
Program  (which  was  formerly  controlled  by  the  Directorate 
for  Lessons  Learned  and  Systems  Support,  ALD/LSL)  and  is 
ALD's  functional  support  that  deals  with  support  equipment 
issues  (for  acquisition  programs).  For  purposes  of  this 
study  only,  the  Air  Force  Lessons  Learned  Program  (AFLLP) 
shall  be  considered  LSE's  main  responsibility. 

The  mission  of  the  AFLLP,  as  stated  in  a  1991  AFLLP 
"educational"  packet,  is  (12): 

(a)  "Gather  experiences  from  all  Air  Force  Major 
Commands  and  Airmy  and  Navy  lessons  learned  programs;" 

(b)  "Validate  these  experiences;" 
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(c)  "Maintain  a  corporate  memory  bank  of  experiences 
as  lessons  learned;" 

(d)  "Make  the  lessons  available  to  both  government  and 
industry  personnel  through  on-line  access,  tailored  lessons 
learned  packages  and  the  Abstract." 

The  AFLLP  considers  a  "lesson  learned"  to  be  a 
"recorded  experience  of  value  in  conducting  current  or 
future  programs  or  modifications,"  where  such  an  experience 
can  be  positive  or  negative  in  nature  (13).  Therefore,  any 
experience  that  can  indicate  a  more  efficient  or  effective 
way  of  accomplishing  a  task  (that  is  related  to  the 
acquisition  of  DOD  weapons  systems  or  supporting  systems)  is 
a  potential  AFLLP  "lesson  learned." 

For  many  of  its  approximately  fourteen  years  of 
existence  (13),  the  AFLLP  had  acquired  "lessons  learned" 
primarily  from  "logistics-oriented”  USAF  agencies,  such  as 
the  Air  Logistics  Centers  (ALCs) .  This  is  not  unexpected, 
given  the  orientation  of  ALD  and  its  parent,  AFLC. 

However, in  recent  years,  it  has  become  clear  that  "lessons 
learned"  from  system  developers  (both  government  and 
contractor)  and  users  such  as  Tactical  Air  Command,  or  TAC, 
could  be  used  "up  front"  to  improve  the  development, 
acquisition,  and  deployment  of  future  weapon  and  support 
systems.  So,  as  discussed  in  a  1991  AFLLP  background  paper, 
one  of  the  objectives  of  the  current  AFLLP  is  to  document 
"the  experiences  of  users  and  maintainers  of  systems  and 
equipment"  and  to  feed  "this  information  back  to  the 
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designers”  (13) .  Therefore,  ALD/LSE  interfaces  with  "user” 
agencies,  "developer”  agencies,  "acquisition”  agencies,  and 
"logistics”  agencies,  as  part  of  the  AFLLP. 

The  AFLLP  is  currently  authorized  by  a  joint 
regulation,  AFLC/AFSC  800-37,  dated  7  August  1981.  This 
regulation  can  be  supplemented  by  the  various  AFSC  and  AFLC 
agencies;  for  example.  Aeronautical  Systems  Division  (in 
AFSC)  has  a  supplement  dated  15  Apr  1988.  The  regulation 
(and  the  supplements)  describe  the  general  responsibilities 
of  the  various  organizations  at  the  "command  staff”  or 
"product  division”  level  (or  "ASD  staff”  or  'two  letter" 
level,  in  the  ASD  supplement)  to  support  the  AFLLP  (14,  15). 

One  area  outlined  in  the  regulation/ supplement  concerns 
the  "interfaces”  between  the  AFLLP  OPR  (now  ALD/LSE)  and  the 
various  AFLC  and  AFSC  agencies.  In  order  to  promote 
awareness  and  implementation  of  the  AFLLP  within  the  "lower 
level"  organizations,  each  acquisition-related  program 
office  is  expected  to  have  a  "Lessons  Learned"  focal  point. 
The  appointed  individual  is  supposed  to  encourage 
development  of  "lessons  learned"  from  within  his  or  her 
agency,  and  to  support  his  or  her  program  co-workers  by 
helping  them  to  obtain  "lessons  learned"  relevant  to  their 
efforts  (14,  15).  Such  focal  points  are  now  also  being 
established  at  various  non-AFLC/AFSC  agencies,  such  as 
Strategic  Air  Command  (SAC)  and  Air  Training  Command  (ATC) 
(16)  . 
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The  AFLLP  OPR  assists  the  focal  points  (and  their 
organizations)  by  providing  detailed  explanations  of  the 
AFLLP  mission,  AFLLP  information  "needs"  and  the  procedures 
for  submitting  (and  obtaining)  "lessons  learned." 

Educational  packets,  brochures,  posters,  office  supplies 
(labelled  with  the  phrase  "Lessons  Learned"  and  the  "name 
and  phone  #"  of  the  AFLLP  program) ,  and  similar  materials 
are  provided  by  ALD/LSE  to  the  "focal  points",  and  other 
interested  individuals  and  agencies.  In  addition,  the 
program  manager  for  AFLLP  often  travels  to  various  agencies 
to  brief  current  (or  prospective)  AFLLP  participants  about 
the  program  (17) .  The  LSE  staff  also  write  articles 
describing  the  program,  for  various  periodicals  such  as  the 
CSERIAC  GATEWAY  (for  information  analysts)  (18). 

The  AFLLP  "cycle"  for  "lessons  learned"  can  be  looked 
at  as  having  three  phases:  submission  of  potential  lessons 
learned,  validation  of  those  potential  lessons  and 
"acquisition"  from  the  AFLLP  OPR's  database. 

In  the  submission  phase,  potential  "lessons  learned" 
are  created,  using  as  references  the  formats  and  "tips" 
provided  in  the  AFLLP  Lessons  Learned  Writing  Guide  (or 
other  AFLLP  educational  material  with  similar  information) . 
The  format  required  is  the  same  as  shown  on  AF  Form  1521, 
"Potential  Lessons  Learned  Submittal  Record,"  Jan  90  (See 
Appendix  B) ,  although  the  form  itself  is  optional  (19).  The 
"addressee"  is  ALD/LSE,  while  the  "sender"  is  the  submitting 
agency;  the  name  of  the  individual  submitter,  his  or  her 
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telephone  number,  and  the  date  submitted  are  to  be  shown,  in 
case  more  information  on  the  subject  is  desired  (either  by 
ALD/LSE,  the  validators,  or  later  lesson  "customers'*  who 
wish  to  know  more  "background"  (17) . 

The  "layout"  of  the  actual  "information"  sections  is  as 
follows  (19) : 

Topic: 

Lesson  Learned: 

Problem: 

Discussion: 

Recommended  Action: 

The  "topic"  describes  the  subject  of  the  lesson 
(usually  two  lines  maximum) .  The  "Lesson  Learned"  is 
supposed  to  describe  in  one  or  two  sentences  the  "single 
most  important  finding,"  which  must  exhibit  "a  cause  and 

I 

effect  relationship"  such  that  a  reader  can  understand  the 
"driver"  and  "results"  of  the  "scenario"  depicted  in  the 
lesson  (19).  The  "Problem"  is  supposed  to  describe  the 
actual  circumstances  of  "what  went  wrong"  in  one  or  two 
succinct  sentences;  if  the  lesson  is  positive,  this  is 
"None"  (19) .  The  "Discussion"  provides  the  background  and 
supporting  information,  usually  one  to  three  paragraphs 
needed  to  explain  the  situation (s)  that  had  "brought  about 
the  findings."  The  "Recommended  Action"  is  supposed  to 
identify  "what"  should  be  done,  "who"  should  be  doing  it, 
and  "when"  during  the  phases  of  an  acquisition  program  the 
action  should  be  done  (19).  Assuming  that  this  potential 
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lesson  is  "beneficial”  -  someone  reading  would  indeed  follow 
a  positive  action  and  avoid  a  negative  action,  "valid"  -  the 
result  of  a  real  experience  and  not  just  a  theoretical 
possibility,  and  "applicable"  -  does  not  require  infeasible 
or  unclear  actions  -  then  it  is  submitted  to  ALD/LSE, 
normally  in  a  "paper"  copy  (19)  or  through  the  electronic 
"on-line"  access  to  the  AFLLP  database  (20) .  Any  supporting 
documents  that  are  used  to  substantiate  the  potential 
"lessons  learned"  are  also  supposed  to  be  sent  (usually  on 
paper  copies)  to  ALD/LSE,  along  with  the  potential  Lesson 
Learned  (21) . 

In  the  second  phase  of  the  AFLLP  cycle,  the  ALD/LSE 
staff  attempt  to  "validate"  the  potential  lessons  learned. 

In  this  phase,  "validation"  refers  to  (22) : 

(a)  The  determination  of  whether  or  not  the  potential 

I 

"lesson  learned"  is  already  adequately  described  in  the 
AFLLP  database  or  some  other  "readily  accessible" 
information  source  (such  as  "Military  Standards",  DOD  or 
USAF  regulations,  and  similar  items),  and  if  not,  then 

(b)  The  review  and  revision  (as  needed)  of  the 
potential  "lesson  learned,"  to  meet  format/content 
requirements  or  to  "improve  the  clarity"  of  the  specific 
lesson  (and  sometimes  other  lessons  as  well) . 

Since  many  potential  "lessons  learned"  concern  subjects 
for  which  the  ALD/LSE  staff  are  not  the  "cognizant  experts," 
the  staff  obtains  the  assistance  of  other  agencies  to 
validate  potential  "lessons  learned."  These  agencies 
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include  user,  developer,  acquisition  and  logistics  agencies, 
each  of  which  provides  a  "validator"  to  support  the  LSE 
staff  in  certain  specified  subject  areas  (23) .  These 
validators  use  the  "Lessons  Learned  Validator's  Guide" 
(provided  by  the  LSE  staff)  as  their  reference  in  reviewing 
potential  lessons.  The  “Validator's  Guide"  explains  the 
format  and  criteria  for  "lessons  learned"  and  provides  the 
"Lessons  Learned  Validation  Worksheet"  (AFLC  Form  8015,  Aug 
90)  and  instructions  for  completing  that  form,  plus  a 
checklist  for  confirming  the  proper  completion  of 
validation  review  (22) . 

The  validator  is  expected  to  provide  one  of  three 
recommendations  (22) : 

"Lesson  Learned"  -  approved  "as  is," 

"No  Lesson  Learned"  -  not  suitable, 

"Lesson  Rewrite"  -  required  revision. 

Each  recommendation  must  be  explained  in  detail  on  the  Form 
8015  (and  a  revised  Lesson  attached,  if  applicable) ,  so  that 
the  LSE  staff  will  understand  the  reasons  for  the  decisions. 
In  all  cases,  the  validator's  name  is  included  on  the  form, 
so  that  additional  information  can  be  obtained  if  needed 
(22) .  Approved  and  rewritten  "potential"  lessons  learned 
are  now  actual  "Lessons  Learned"  and  are  placed  in  the 
databases  (17). 

In  the  third  phase  of  the  AFLLP  "cycle, "  prospective 
users  of  lessons  learned  interface  with  the  AFLLP  staff  and 
database  to  acquire  the  desired  information. 
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Dealing  through  the  AFLLP  staff  is  the  first  step  in 
this  phase.  The  staff  must  check  the  affiliation  of  the 
“Lessons  Learned”  requester  to  determine  if  that  requester 
is  eligible  for  access  to  the  AFLLP  “Lessons  Learned."  This 
check  is  necessary  because  (although  unclassified)  many  of 
the  “lessons  learned"  contain  information  that  is  sensitive, 
or  subject  to  the  various  "export  control"  laws  such  as 
those  in  Public  Law  98-94  (24) .  If  the  requester  is 
confirmed  to  be  a  U.S.  Government  employee,  eligibility  is 
granted  (17) .  Non-government  agencies  or  individuals,  such 
as  DOD  contractors,  must  file  a  Militarily  Critical 
Technical  Data  Agreement  (DD  Form  2345)  with  the  Defense 
Logistics  Agency  (DLA) .  Once  the  contractor  has  forwarded  a 
copy  of  the  DLA-certif ied  DD  Form  2345  to  ALD/LSE,  that 
contractor  is  considered  eligible  for  the  AFLLP  (24). 

Once  the  eligibility  of  the  requester  is  established, 
the  requester  can  make  written  or  verbal  requests  to  the 
staff.  These  requests  can  ask  for  a  paper  copy  of  the 
Abstract  of  Lessons  Learned  volume  (published  yearly  by 
ALD/LSE) ,  or  for  a  Lessons  Learned  search  about  the 
appropriate  subjects  of  interest.  The  requests  can  also  ask 
for  a  password  and  account  for  the  AFLLP 's  electronic 
database,  so  that  the  requester  may  perform  his  or  her  own 
Lessons  Learned  search  (17).  Staff-performed  searches  are 
probably  most  useful  for  requesters  who  don't  have  access  to 
"electronic  mail  &  Defense  Data  Network"  systems  (25)  or 
computer/modem  systems,  or  for  those  who  don't  mind  waiting 
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several  days  for  their  lessons  (since  the  search  results  are 
sent  via  the  mail) .  However,  many  "would-be”  Lessons 
Learned  users  apparently  prefer  to  try  the  "on-line,"  self¬ 
search  option  (21) . 

Once  the  "on-line"  searcher  has  gotten  into  the  AFLLP 
database,  the  software  offers  three  main  methods  for 
locating  Lessons  Learned.  These  methods  are  as  follows 
(20)  ; 

(a)  2  or  3  word  string  search;  if  this  method  is 
chosen,  the  searcher  must  designate  the  "impact  area"  and 
"program  phase"  of  the  subject  of  interest,  then  the  2  or  3 
words  denoting  the  desired  subject.  The  "impact  areas"  are 
currently  67  in  number  and  reflect  various  categories  of 
interest  (such  as  program  management,  reliability,  and 
others) ,  while  the  "program  phase"  refers  to  the  stage  in 
the  acquisition  life  cycle  of  a  weapon  system  (concept 
exploration,  demonstration/validation,  and  so  forth) .  The 
word  string  is  supposed  to  be  the  specific  topic  of 
interest;  it  must  be  in  words  in  consecutive  order,  and 
excludes  certain  words  (such  as  "of,"  for  instance); 

(b)  Retrieve  by  Lesson;  this  method  uses  the  indexing 
Lesson  numbers,  and  can  allow  the  searcher  to  review  the 
whole  lesson  or  just  certain  parts  (such  as  the  abstract) ; 

(c)  Independent  Word  Search;  this  method  works 
similarly  to  the  "string"  search,  except  that  the  1,  2,  or  3 
words  sought  for  can  be  located  anywhere  within  the  lesson. 


35 


The  search  software  is  designed  to  record  the  identity 
of  the  searcher,  as  well  as  the  number  of  searches  performed 
and  the  specific  Lessons  Learned  that  were  found/retrieved 
(21) .  The  AFLLP  staff  are  thus  able  to  use  certain 
"quantitative”  measurements  of  their  AFLLP  activities,  such 
as  the  total  number  of  LL  requests  over  a  specific  period  of 
time  (26) .  In  addition,  certain  "qualitative"  measurements, 
such  as  the  identities  of  specific  searchers  and  the  lessons 
they  found,  can  be  shown  -  if  a  staff  member  manually 
reviews  and  interprets  the  recorded  information  (21) . 

Another  qualitative  measurement  "metric"  used  for  the 
AFLLP  is  "customer  feedback."  This  feedback  ranges  from 
informal  comments  made  to  ALD/LSE  during  "education" 
briefings,  to  detailed  comments  provided  by  "on-line" 
searchers  -  a  section  of  the  AFLLP  database  software  allows 
the  searcher  to  provide  comments  for  the  staff  prior  to 
logout  (20) .  In  addition,  the  staff  sends  out  a  short 
questionnaire  with  mailed  Lessons  Learned  packages;  the 
questionnaire  asks  the  customer  to  rate  the  staff  service 
and  lessons  learned  (Excellent-Good-Fair-Poor) ,  and  requests 
the  customer  provide  specific  comments  concerning  the 
helpfulness  and  usefulness  (or  lack  thereof)  of  the  lessons 
(27) .  The  questionnaire  is  not  currently  available  to  "on¬ 
line"  searchers  (20,  21)  . 

The  AFLLP  staff  is  currently  considering  several 
changes  aimed  at  improving  the  program;  some  are  in  active 
development.  For  example,  ALD/LSE  personnel  have  held 
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discussions  with  the  various  Army  and  Navy  Lessons  Learned 
personnel,  to  discuss  possible  options  for  improved 
integration  of  the  different  service  databases  (21,  25) . 
Second,  the  AFLLP  is  pursuing  development  of  a  "personal 
computer"  version  of  the  AFLLP  database  system,  which  is 
desirable  to  many  agencies  (among  them  classified  programs) 
who  find  the  telephone-linked,  "on-line"  system  impractical. 
Third,  ALD/LSE  staff  are  attempting  to  obtain  an  improved 
version  of  their  Oracle  "search  and  retrieve"  software 
(currently  part  of  the  AFLLP  database  system) ,  to  provide 
such  features  as  "phrase  search"  (21) . 

AS D/TO:  The  mission  of  ASD/TQ  is  -  basically  -  to 

"promote  Total  Quality  (TQ)  throughout  ASD"  by  "assisting 
leaders  in  developing  TQ  attitudes  and  initiatives  with  ASD 
customers  and  suppliers"  and  acting  as  "catalysts  for 
cultural  change"  to  encourage  TQ  adoption  (28) .  Total 
Quality  is  defined  by  ASD  as  "a  leadership  philosophy  that 
creates  a  working  environment  which  promotes  trust, 
teamwork,  and  the  quest  for  continuous  improvement,"  and  has 
as  its  overall  objective  "(the)  deliver (y)  (to)  the  user 
(of)  a  product  that  consistently  performs  to  correctly 
defined  requirements  that  satisfies  his  needs  and 
expectations"  (28) . 

In  order  to  accomplish  this  mission,  ASD/TQ  works  with 
all  USAF  agencies  in  ASD,  as  well  as  with  the  contractors 
who  support  ASD  programs  (the  "suppliers") ,  and  the  "users" 
of  ASD's  products  (the  "customers"),  such  as  TAG  and  MAC. 
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ASD's  TQ  plan  was  approved  in  1988,  by  the  ASD 
Commander,  then  Lieutenant  General  John  Loh  (29) ;  it  was 
based  upon  promising  results  obtained  in  1987  from  projects 
initiated  by  two  ASD  agencies,  the  Propulsion  and  Training 
Systems  SPOs  (30) .  A  contractor,  the  Cumberland  Group,  was 
"brought  on  board”  to  train  and  advise  ASD  agencies  in  the 
proper  development  of  TQ  structures,  tools  and  techniques 
(29)  . 

Each  organization  within  ASD  has  adopted  a  "TQ 
structure;"  that  structure  contains  the  following  groups 
(31)  ; 

(a)  a  Steering  Committee,  made  up  of  "upt^er 
management"  within  the  organization; 

(b)  a  TQ  Team,  made  up  of  "a  cross  section"  of 
organization  members,  who  "establish  and  operate"  four 
subteams  needed  to  implement  TQ.  There  are  subteams  for: 
Education  (of  the  organizations,  to  support  TQ) ;  "Search  for 
Opportunities  (SFOs),"  which  "capitalizes  on  employee¬ 
generated  ideas  for  improvements;"  Corrective  Action  (which 
"uses  Corrective  Action  Teams  to  solve  problems  identified 
by  the  employees") ;  and  Measurement  ("to  gauge  progress") ; 

(c)  Corrective  Action  Teams  (CATs) ,  which  are 
commissioned  by  a  CA  subteam,  or  the  Steering  Committee,  to 
"work  specific  problems  within  an  organization;" 

(d)  Critical  Process  Teams  (CPTs) ,  which  are 
"chartered"  by  the  Steering  Committee  to  "investigate  high 
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level,  cross-organizational  processes  which  have  a  critical 
impact  on  satisfying  the  customers'  requirement." 

At  the  ASD  level,  there  is  an  ASD  Executive  Steering 
Committee  (ESC) ,  "chaired  by  the  ASD  Commander"  (or  Vice 
Commander)  and  "with  a  cross  section  of  ASD  top  management 
as  its  membership"  (31)  that  addresses  TQ  issues  and 
policies  at  "ASD-wide"  or  "outside  ASD"  levels.  For 
example.  Category  III  -  "outside  a  program  office 
applicability"  (32)  -  SFOs  must  be  reviewed  and 
"prioritized"  by  the  ASD  ESC;  another  of  the  ASD  ESC's 
responsibilities  is  to  "Provide  a  Forum  for  Lessons 
Learned/Cross  Talk"  (28) . 

The  ASD/TQ  office  acts  as  a  "facilitator"  for  many  of 
the  "ASD-wide"  or  "outside  ASD"  actions  dealt  with  by  the 
individual  organizations  and  the  ASD  ESC.  For  example,  each 
cognizant  "subteam"  facilitator  within  ASD/TQ  chairs  the 
periodic  meetings  of  the  four  subcommittees  -  whose  members 
are  the  individual  organization  "focal  points"  for  that 
particular  type  of  subteam  -  all  SPO  Education  subteam  focal 
points  would  attend  a  monthly  Education  subcommittee  Forum, 
for  instance  (33). 

In  these  "specific"  Forums,  the  focal  points  discuss 
their  agency's  TQ  activities,  and  in  particular  the 
activities  they  feel  would  have  relevancy  outside  their 
specific  program  office/other  agency.  Most  activities 
concern  "quality  of  life"  issues  -  such  as  workplace 
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such  as 


environment  -  or,  increasingly,  "process  issues" 
shortening  Statement  of  Work  coordination  time  (33) . 

ASD/TQ  also  "facilitates"  the  TQ  Open  Forum,  in  which 
"anyone”  (usually  members  from  the  four  Subcommittees)  can 
discuss  or  "find  out  about”  various  types  of  TQ  issues  (33) . 

ASD/TQ  does  have  a  specifically  identified  type  of 
"lessons  learned”  effort,  which  is  known  as  "Wins."  ASD/TQ 
considers  a  "Win”  to  be  any  "proven  to  work"  course  of 
action  or  accomplishment  that  should  (have) ,  when  properly 
implemented,  result (ed)  in  an  improved,  successful  "way  of 
doing  business"  (33,  34);  this  is  essentially  a  "positive- 
oriented"  analog  to  "lessons  learned."  Currently,  the 
"Wins"  effort  is  basically  an  informal  method  of  furnishing 
"success"  stories  to  different  groups  within  ASD.  The 
"Wins"  process  works  as  follows  (33) : 

(a)  ASD/TQ  is  provided  descriptions  of  potential 
"Wins,"  along  with  supporting  information  (to  explain  why 
the  specific  action/process/accomplishment  should  be 
considered  a  "Win") ; 

(b)  If  the  cognizant  ASD/TQ  "facilitators"  agree  with 
the  rationale,  the  submitter  is  notified  that  he  or  she  can 
"brief"  that  item,  as  a  "Win,"  at  the  appropriate  Forums  (as 
determined  by  the  facilitators) .  In  certain  cases  ASD 
Executive  Steering  Committee  approval  may  be  required,  if 
the  "potential  Win"  deals  with  issues  outside  "normal" 
program  office  cognizance  (such  as  Category  III  SFOs,  for 
example) ; 
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(c)  The  submitting  agency  describes  and  explains  the 
"Win,"  during  a  presentation  given  at  the  appropriate 
Forums.  Copies  of  the  briefing  charts  (or  other  media)  are 
provided  to  ASD/TQ  and/or  kept  by  the  "Win"  submitter. 

ASD/TQ  obtains  most  of  its  information  about  ASD  TQ 
activities  from  the  subteam  focal  points,  the  ASD/ESC, 
different  contractor  agencies  (the  Cumberland  Group,  for 
example,  is  still  providing  information  on  basic  TQ  methods 
and  techniques) ,  and  the  many  CAT  and  CPT  teams.  ASD/TQ 
requests  for  information  are  often  handled  using  telephone 
or  personal  contact  procedures,  or  written  letters; 
responses  are  usually  provided  in  a  similar  matter.  Any 
SFO,  CAT,  or  CPT  inputs  are  provided  in  written  paper  form 
(33) .  The  "SFO"  writeups  are  supposed  to  be  one  page 
"summaries"  of  the  proposed  idea,  while  CAT  results  are 
generally  short,  "background  paper"  submittals,  and  CPT 
findings  are  usually  equivalent  to  "Technical  Reports"  in 
their  level  of  detail  (34). 

ASD/TQ  personnel  usually  make  few  (or  no)  changes  to 
the  submitted  material  -  this  practice  is  intended  to 
discourage  the  "filtering"  of  information  that  may  result 
from  the  information  "passing  through  many  hands"  (34) . 

The  ASD/TQ  office  maintains  certain  types  of  this 
information  "on  file"  -  such  as  educational  or  informative 
material,  "self-generated"  SFOs,  "current  issues  in  work" 
documents,  and  similar  types  of  data  -  but  in  many  cases  the 
"documented"  information  is  kept  by  other  organizations 
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(quite  often  the  submitter  or  the  "responsible"  agency) .  If 
maintained  by  ASD/TQ,  it  is  often  kept  on  "paper,"  magnetic 
computer  disks,  or  personal  memories;  reports,  working 
notes,  and  briefing  charts  are  the  most  common  types  of 
"documents"  used  (33,  34) . 

ASD/TQ  does  keep  track  of  the  type  and  disposition  of 
"Category  III"  SFOs;  they  use  a  "status"  list,  which 
describes  the  SFOs  by  title,  key  word,  log  number, 
organization,  and  short  (ten  words  or  less)  status  and 
"resolved"  comments  (35) .  This  list  is  maintained  on 
magnetic  disk  and  in  written  paper  form,  and  can  be  reviewed 
by  other  agencies  through  communication  with  ASD/TQ.  The 
specific  SFO  can  be  retrieved  from  the  "list"  using  "key 
word"  search  or  log  number  if  reviewing  on  a  computer,  or  by 
those  and  title  if  reviewing  paper  copies  (34) . 

When  ASD/TQ  disseminates  information,  it  uses  different 
methods  and  forms,  depending  upon  the  "audience"  and  the 
subject  matter  of  the  information.  For  example,  when  ASD 
management  decided  ASD  should  compete  for  the  1990  Quality 
Improvement  Prototype  (QIP)  award,  inputs  from  each  "two- 
letter"  organization  were  reviewed  -  by  a  multi-organization 
"tiger  team"  lead  by  ASD/TQ  (36)  -  reformatted,  edited,  and 
put  into  a  "booklet"  entitled:  "TQ:  A  Journey  to 
Excellence,"  which  described  ASD  and  explained  its 
accomplishments  in  implementing  TQ  principles  and  practices 
(37).  This  "booklet,"  as  well  as  an  earlier  "booklet" 
created  for  the  1989  QIP  competition,  were  designed  to  be 
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read  by  the  "public  at  large,"  and  were  widely  disseminated 
(34)  . 

When  ASD/TQ  provides  information  in  "education 
workshops"  with  government  or  industry  agencies,  the  form  is 
often  personal  discussions,  with  paper  charts  and/or 
audiovisual  presentations,  and  is  usually  confined  to  the 
use  of  the  "workshop"  participants,  until  they  return  to 
their  agencies  (33) . 

Most  TQ  information  can  be  provided  to  requesting 
industry  or  government  representatives,  although  in  many 
cases  (such  as  SFO  review)  information  is  not  released 
outside  ASD  until  a  decision  (concerning  the  issue  in 
question)  has  been  announced  (33) . 

Metrics  are  always  a  concern  in  ASD/TQ  -  hence  the 
concentration  on  Measurement  subteam  activities  for  each  TQ 

I 

issue.  The  specific  types  of  metrics  are  dependent  upon  the 
specific  situation,  although  many  metrics  that  attempt  to 
measure  "results"  are  "quantitative"  in  nature.  One  such 
example  is  measuring  the  number  of  clarification  requests 
given  to  contractors  (about  their  responses  to  a  Request  for 
Proposal)  before  and  after  TQ  tools  and  information  have 
been  provided  to  the  contractors  (28)  .  Another 
"quantitative"  example  is  the  analysis  of  "critiques" 
answered  by  participants  in  "ASD-Industry  Total  Quality 
Seminars,"  measuring  numbers  of  critics  and  their  responses 
to  standardized  "scaled"  questions  (28) .  One  simpler  metric 
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involves  measuring  the  time  taken  from  "issue  introduction" 
to  resolution,  to  compare  "old"  and  "new"  processes  (28) . 

Many  changes  are  currently  planned  or  "in  process"  at 
ASD/TQ.  Some  of  them  include: 

(a)  An  ASD-wide  "TQ  Newsletter,"  which  would  profile 
TQ  "issues  of  interest"  and  ASD*s  "TQ  progress"  (being 
implemented)  (32,  28); 

(b)  Formalizing  the  "Wins"  process,  through 
development  of  explicit  criteria  (for  deciding  what  a  "Win" 
is)  and  the  use  of  an  electronic  database  to  access,  store, 
and  retrieve  "Wins"  descriptions  (project  in  "concept 
exploration"  stage)  (33) ; 

(c)  "Gearing  up"  to  develop  a  "booklet"  for  the  1991 
QIP  competition  (34) ; 

(d)  Developing  a  cataloging  system  for  CAT/CPT  paper 

I 

reports  (and  maybe  SFOs)  (under  consideration)  (34). 

Provider  Agencies:  The  agencies  in  this  sample 
subdivision  are  the  Headquarters  Air  Force  Systems  Command, 
Deputy  Chief  of  Staff  for  Requirements,  Directorate  of 
Program  Management  (HQ  AFSC/XRM)  and  the  Aeronautical 
Systems  Division,  Deputy  Chief  of  Staff  for  Program 
Management  (ASD/CY) . 

HO  AFSC/XRM;  The  mission  of  HQ  AFSC/XRM,  as 
stated  in  a  1990  XR  internal  planning  paper,  is  "to  create, 
develop  and  integrate  acquisition  policy  and  procedures;  to 
support  the  program  manager  by  providing  professional 
development,  acquisition  tools  and  expert  counsel;  and  to 
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enhance  the  program  management  envi ■ onment  by  continually 
improving  the  acquisition  process,  fostering  teamwork  and 
eliminating  unnecessary  rec  ’irements"  (38) . 

To  support  such  a  mission,  this  XRM  office  is  able  to 
work  with  a  variety  of  organizations  inside  and  outside  HQ 
AFSC,  among  them  the  Systems  (sometimes  called  "Product") 
Divisions,  the  Test  Centers,  AFLC  agencies,  "functional" 
offices  such  as  HQ  AFSC/EN  (DCS  for  Engineering)  and  HQ 
AFSC/PK  (DCS  for  Contracting) ,  the  Program  Executive 
Officers  (PEOs) ,  and  SAF/AQ,  the  support  office  for  the 
Assistant  Secretary  of  the  Air  Force  for  Acquisition  (39) . 

HQ  AFSC  believes  the  "Lessons  learned"  phrase  is 
generally  more  closely  associated  with  "experiences  to  avoid 
repeating,"  and  therefore  has  related  AFSC's  definition  of 
"Lessons  Learned"  to  "past  mistakes"  (39,  40) .  Positive 
experiences,  particularly  those  with  the  potential  to 
"significantly"  improve  the  acquisition  process,  are 
considered  to  be  "best  practices"  (41) .  The  XRM  office  is  a 
focal  point  for  HQ  AFSC  programs  dealing  with  both  "Lessons 
Learned"  and  "best  practices."  The  "Lessons  Learned" 
efforts  are  generally  handled  within  XRMA,  while  the  "best 
practices"  projects  are  handled  within  XRMP;  XRMA  focuses  on 
the  goal  of  continuously  improving  the  program  management 
process,  while  XRMP  focuses  on  the  goal  of  continuously 
improving  acquisition  policy  (38) . 

XRMA:  This  organization  acts  as  the  HQ  AFSC  focal 

point  for  the  AFSC  Program  Management  Lessons  Learned 
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Process  (PMLLP) .  The  process  is  intended  to  "tap”  the 
knowledge,  experience  and  insights  of  current  acquisition 
program  managers  (and  other  acquisition  "sources  of 
expertise") ,  provide  that  information  to  other  acquisition 
managers  on  a  "realtime"  basis,  and  provide  a  database  of 
"lessons  learned"  for  program  directors  and  others  to  use 
(39)  . 

The  PMLLP  was  begun  in  early  1991,  in  response  to 
concerns  expressed  during  the  October  1990  Program  Directors 
(PDs)  Conference  (42) .  It  is  outlined  in  written  and 
diagram  form  in  AFSCR  550-19,  dated  28  June  1991  (which  is 
shown  in  Appendix  B) . 

The  key  features  of  the  PMLLP  are  as  follows: 

(a)  HQ  AFSC/XR  has  created  a  set  of  electronic 
"information  networks,"  or  INFONETS,  and  have  made  them 
available  for  use  by  the  various  SPO  program  directors,  HQ 
AFSC  functional  offices,  and  the  AFPEO  and  SAF/AQ  agencies. 
One  of  these  networks  is  the  monthly  Program  Directors  (PD) 
INFONET,  which  iS  intended  primarily  to  pass  current  "IG" 
and  management  information  of  interest  from  PDs/AFSC  to 
AFSC/PDs  (43).  The  other  is  the _ Program  Management  Lessons 
Learned  INFONET,  which  is  intended  to  provide  "Lessons 
Learned"  by  individual  PDs  (and  AFSC  offices,  PEOs  and 
SAF/AQ  agencies)  to  the  rest  of  their  fellows  in  the 
acquisition  field,  on  a  "current"  timeline  (42) . 

(b)  The  PM  Lessons  Learned  are  to  be  submitted  to  XRMA 
in  a  format  similar  to  that  used  for  the  Air  Force  Lessons 
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Learned  Program  (AFLLP)  -  one  or  two  page(s),  main  "Lesson, 
brief  explanation  and  recommendation  (44)  -  except  that  no 
editing  (by  AFSC)  or  validation  is  required  (39) . 

(c)  AFSC/XRMA  retransmits  these  "Lessons"  (minus  only 
the  "originator"  ID)  over  the  PMLL  INFONET,  and  sends  the 
"Lessons"  into  the  Program  Directors  "impact  area"  of  the 
AFLLP  database  (39). 

(d)  The  Systems  Divisions  are  tasked  to  set  up  "face- 
to-face  meetings  between  program  offices,"  to  provide  for 
other  avenues  to  distribute  "Lessons  Learned"  (40) . 

(e)  The  PMLL  INFONET  or  "face-to-face"  meetings  should 
be  used  to  disseminate  key  information,  such  as  "lessons" 
resulting  from  major  program  events,  or  the  performance  of 
contractors  that  are  "common"  to  multiple  program  offices, 
or  similar  important  knowledge  (40) . 

(f)  The  AFSC  PMLLP  is  intended  primarily  for 
"executive-level"  Lessons  Learned  (45) ;  lower  level 
management  is  encouraged  to  "support  the  Air  Force  Lessons 
Learned  Program"  (40) . 

In  support  of  this  PMLLP,  AFSC/XRMA  has  been  performing 
several  activities.  One  is  drafting  a  revision  to  the  Air 
Force's  "program  management"  regulation,  AFR  800-2.  This 
revision  would  add  a  "Section  H"  to  Part  5  of  that 
regulation,  dealing  with  "Air  Force  Lessons  Learned,"  that 
would  outline  the  "policy  and  procedures  to  identify, 
document,  and  publish  acquisition  and  operational  lessons 
learned;"  this  section  would  also  identify  ALD/LSE  as  the 
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OPR  for  the  AFLLP,  and  would  define  "lessons  learned"  in  the 
broader  "positive  and  negative"  sense  currently  used  by 
ALD/LSE  (46)  . 

A  significant  concern  about  the  PMLLP  is  "metrics." 
Currently,  XRMA  is  able  to  review  the  number  (and  source 
agency)  of  "PM  Lessons  Learned"  submitted,  and  the  number 
and  identity  of  "addresses"  for  the  PMLL  INFONET.  XRMA  has 
requested  support  for  and  feedback  about  the  PMLLP,  using 
both  the  PD  and  PMLL  INFONETS  (43,  45).  Any  comments  or 
feedback  (and  quantitative  measures  such  as  number  of 
searchers)  obtained  about  the  "Program  Directors'  impact 
area  Lessons  Learned"  can  be  provided  to  XRMA  from  ALD/LSE, 
upon  request  (47) . 

One  set  of  concerns  expressed  by  certain  agencies  was 
that  the  PMLLP  should  be  expanded  to  include 
"other .. .disciplines"  besides  program  management  (48).  In 
response,  XRMA  has  been  investigating  the  "ET-NET" 
engineering  and  technology  electronic  information 
database  -  run  by  AFSC/EN;  the  "TEC-NET"  joii  "  test  and 
evaluation  electronic  database  -  run  by  the  Navy  at  Patuxent 
Naval  Air  Station  in  Maryland;  the  Navy  Lessons  Learned 
program;  and  the  various  Army  Lessons  Learned  efforts  (39) . 

Initial  findings  from  those  investigations  indicate: 

(a)  The  "functional"  databases  (ET-NET  and  TECH-NET) 
currently  have  no  provisions  for  "lessons  learned"  inputs 
(or  outputs)  (49) ; 
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(b)  The  Army  efforts  are  very  fragmented  at  present, 
with  different  commands  (and  a  number  of  different  agencies 
within  those  commands)  having  responsibility  for  different 
types  of  "subject”  databases  (50) ; 

(c)  The  Navy  Lessons  Learned  program  bears  many 
similarities  to  the  Air  Force  program  as  heretofore  run  by 
ALD/LSE  -  "primarily  consists  of  a  database,"  Lessons 
"appear  to  focus  on  one  area"  (systems  engineering) ,  format 
is  "almost  identical  to  the  AF"  -  except  that  the  Naval  Air 
Systems  Command  personnel  responsible  for  the  program 
actually  write  the  "lessons  learned"  (based  upon  inputs  from 
"field  units")  and  validate  it  at  the  same  time  (51) ; 

(d)  Although  the  Navy  currently  does  provide  some  of 
its  "lessons  learned"  to  the  ALD/LSE  database,  there  is  not 
yet  the  sufficient  hardware  and  software  "network  stability" 
needed  to  shift  to  a  joint  ALD-housed  common  database  - 
although  coordination  efforts  are  continuing  (25) . 

Another  change  XRMA  has  recently  made  is  in  the  format 
requirements  for  the  PM  "lessons  learned;"  in  an  effort  to 
encourage  more  submittals,  the  PDs  and  others  may  now 
provide  "lessons"  to  XRMA  in  whatever  form  they  feel 
appropriate,  and  XRMA  will  retransmit  the  LL  in  that  form 
over  the  PMLL  INFONET.  As  for  the  transmittal  into  the 
AFLLP  database,  XRMA  will  rewrite  the  "Lessons"  to  conform 
with  the  AFLLP  format  (52) . 

The  XRMA  office  is  also  considering  new  metrics,  such 
as  a  "customer"  survey  or  questionnaire,  to  get  a  better 
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idea  of  what  users  of  the  PMLLP  (and  perhaps  AFLLP)  "really 
think"  (47) . 

The  PMLLP  program  is  scheduled  for  review  at  the  Fall 
91  PDs  Conference,  to  determine  what  additional  changes 
should  be  made  (to  encourage  greater  participation  or 
discontinue  the  project  if  appropriate)  (52) . 

In  addition,  one  major  "Lessons  Learned"  change  is 
currently  scheduled  for  July  1992:  the  transfer  of 
responsibility  for  the  AFLLP,  from  ALD/LSE  to  the  new 
AFMC/XR  (52) . 

XRMP:  This  office  acts  as  a  focal  point  for 

multiple  acquisition  process  improvement  projects,  many  of 
which  are  derived  from  Acquisition  Process  Excellence  II  - 
which  was  chartered  in  June  1990  by  Lieutenant  General  David 
Teal,  the  Vice  Commander  of  AFSC,  as  a  "follow-on"  to  the 
original  APEX  project  (53) .  One  of  these  efforts  is  the 
"best  practices"  set  of  processes. 

This  "best  practices"  set  of  processes  consists  of  the 
APEX  II  Best  Practice  Process  (APEX  II  BPP)  and  the  AFSC 
Best  Practice  Process  (AFSC  BPP) .  The  APEX  II  BPP  was  the 
"first"  of  the  two  BPPs  -  the  initial  version  is  described 
in  a  January  1991  AFSC/XR  White  Paper  (53) ,  and  has  since 
been  modified  to  become  a  part  of  the  "broader"  AFSC  BPP  - 
which  is  described  in  detail  in  AFSC/XR' s  19  June  1991  White 
Paper  and  APEX  II  Meeting  charts  (54,  55). 

The  basic  features  of  the  AFSC  BPP  are  as  follows: 
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(a)  AFSC/XR's  APEX  II  team  creates  a  "pool/repository” 
(54:22)  of  "neat/tested  neat/proven  neat”  ideas  (55)  which 
were  "developed  throughout  the  conunand"  (54:24)  and  other 
acquisition-related  sources  such  as  SAF/AQ,  schools,  and 
other  services  (55) . 

(b)  The  HQ  AFSC  "functional  leads  (the  owners)  cull" 
from  the  pool  of  ideas  a  list  of  "potential  best  practices" 
(54:22).  The  following  are  the  "top  level"  criteria  used 
(54:22) : 

—  "Must  add  significant/ quantified  value  to  the 
acquisition  process;" 

—  "Must  be  universally  applicable  (capable  of  being 
applied  'across  programs'  as  well  as  'to  major  and  component 
programs'  and  must  apply  to  a  large  number  of  programs,;" 

—  "Must  be  trackable/measurable;" 

—  "Must  be  implementable  within  product  (systems) 
division/program  resources  (manpower/money);" 

—  "Must  not  require  additional  ('net')  manpower." 

(c)  The  "potential  best  practices"  are  prioritized  by 
product  (systems)  divisions/program  offices  (which  may  add 
additional  "best  practices") ,  and  separately  by  the  HQ  AFSC 
functional  offices  (55)  ; 

(d)  An  HQ  AFSC  Steering  Group  selects  which  of  the 
"potentials"  will  be  developed  into  "best  practices,"  and 
determines  whether  the  APEX  II  BPP  or  a  "Functional  Process" 
should  be  used  to  develop  them  (55) ; 
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(e)  If  the  APEX  II  BPP  is  selected,  then  the  "owners” 
are  expected  to  "champion"  their  would-be  "best  practices" 
during  a  briefing  to  a  Process  Improvement  Plan  (PIP) 
Steering  Group;  once  a  "potential"  is  "accepted,"  the  PIP 
Steering  Group  advises  and  guides  the  owners  in  developing 
PIPs  and  helps  them  determine  when  the  "best  practices"  are 
ready  for  implementation  (55). 

(f)  Once  the  "best  practices"  have  completed  their 
APEX  II  BPP  (or  functional  process) ,  they  are  implemented  in 
parts  (or  all)  of  the  command;  policies,  processes,  tools 
and  training  are  all  altered  to  support  the  "best  practices" 
(55)  . 

(g)  As  these  "best  practices"  are  developed  and 
distributed  to  the  various  acquisition  agencies,  metrics 
(for  judging  purposes)  for  each  "best  practice"  are  also 
developed,  as  well  as  "command  standards"  (or  "benchmarks") . 
The  "benchmarks"  are  used  to  "evaluate  current  operations 
and  measure  the  success  of ...  implementation"  (54:23)  of  the 
"best  practice(s)"  within  (an)  organization (s) ;  the 
"benchmarks"  are  developed  by  the  same  "organization (s) 
which  developed  the  associated  best  practices  (to  take 
advantage  of  the  expertise  obtained  in  working  with  that 
process"  (54:23-24). 

(h)  Results  of  these  steps  (the  "best  practices"  and 

their  effects)  are  identified  and  used  in  the  development 
and/or  updating  of  the  relevant  acquisition  process 
"model (s)"  and  database (s)  (55); 
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(i)  The  AFSC  functionals  "will  continue  to  collect" 
the  various  types  of  "neat"  ideas,  and  "enter  them  into  the 
repository,"  while  "the  APEX  II  team  will  develop  mechanisms 
to  make  this  information  available  to  the  field"  (54:24). 

Two  examples  of  "best  practices"  that  have  gone  through 
the  first  "cycle"  of  the  BP  process  are  the  "Request  for 
Proposal  (RFP)  Support  Organizations"  (the  "prototype  best 
practice")  and  the  "Acquisition  Security  Course"  (54:23); 
the  "RFPSOs"  are  intended  to  be  groups  of  "RFP-experienced" 
individuals,  who  assist  SPOs  in  the  efficient  "processing" 
of  RFPs  (41) ,  while  the  "ASC"  is  intended  to  "define  the 
role  and  function  of  acquisition  security"  in  the 
acquisition  process  (54:15). 

HQ  AFSC  uses  various  metrics  to  measure  these  efforts 
at  acquisition  process  improvement  streamlining.  One  such 
metric  is  the  quarterly  tracking  of  PIPs  -  number  of  new 
PIPS,  number  of  open  PIPs,  number  of  completed  PIPs,  and  the 
"significant  accomplishments  associated  with  these  PIPs" 
(54:10-11).  Another  metric,  "development  schedules"  - 
planned  compared  to  actual  -  is  used  to  "measure  the 
efficiency  of  the  execution"  of  the  AFSC  Best  Practices  (and 
other)  efforts  (54:10).  HQ  AFSC  also  tracks  the  different 
"streamlining"  initiatives,  by  subject,  command  level,  and 
status  (54:12-16). 

XRMP  is  continually  attempting  to  improve  the  "best 
practices"  and  other  efforts;  several  changes  are  planned  or 
are  being  implemented.  One  such  change  "being  worked"  is 
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the  development  of  the  Acceptable  Practices  Management 
System  (APMS) .  The  APMS  is  intended  to  support  the  "best 
practices”  database/repository  by  providing  a  means  of 
acquiring,  maintaining,  and  distributing  the  various  types 
of  "neat  ideas,"  which  are  also  known  as  "acceptable 
practices"  (56) . 

The  key  features  of  the  APMS  are  as  follows: 

(a)  "Candidate  acceptable  practices"  are  submitted  to 
HQ  AFSC/XRM  (57) ; 

(b)  XRM  uses  an  APMS  software  package  to  create  a  two- 
paragraph  "synopsis"  and  a  one-page  "detailed  writeup" 
describing  each  candidate;  "submitter  information"  (name, 
agency,  phone  number,  address)  and  "tracking  information" 
(subject  category,  control  numbers  and  "suspense" 
schedules) are  included  in  the  "package"  (57). 

(c)  Each  "candidate  package"  is  routed  to  the  HQ  AFSC 
functional  office  that  is  the  cognizant  "reviewer"  for  that 
particular  category  of  interest  (57) ; 

(d)  The  cognizant  "reviewer"  is  expected  to  evaluate 
its  candidate,  and  to  provide  a  detailed  "rationale  for 
acceptance  or  rejection  of  idea  as  Acceptable  Practice"  to 
XRM,  by  the  preassigned  suspense  date  (57) ; 

(e)  If  the  reviewer  approves  the  idea,  XRM  sends  the 
submitter  a  letter  which  explains  that  the  idea  is  now  an 
"acceptable  practice"  (if  the  idea  is  rejected,  an  alternate 
letter  -  with  a  "rejection  explanation"  -  is  sent  to  the 
submitter)  (57) ; 
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(f)  The  "packages”  for  the  new  "acceptable  practices" 
are  then  placed  in  the  database  (or  "databank")  (57) ; 

(g)  XRMP  would  insure  that  "the  databank  would  be 
accessible  to  the  field  for  their  use  as  desired"  (57)  - 
this  is  currently  planned  to  be  accomplished  by  periodically 
distributing/collecting  magnetic  computer  disks  containing 
the  "acceptable  practices"  -  for  use  with  personal  computer 
(PC)  systems  -  to/ from  the  "field  units"  (56) ; 

(h)  XRMP  (and  other  AFSC  offices)  would  "periodically 
review"  the  collected  acceptable  practices,  to  detennine  if 
any  of  these  "effective  methods  for  accomplishing  a  specific 
task  or  series  of  tasks"  might  be  effective  enough  to 
qualify  as  a  "best  practice, ". which  is  "command's 
•preferred'  way  of  accomplishing  a  task"  (57) .  "Promising" 
acceptable  practices  would  be  written  up  as  "potential  best 
practices,"  and  would  enter  the  AFSC  BPP  (56). 

XRMP  is  considering  which  of  many  types  of  metrics 
would  be  appropriate  for  the  APMS.  One  quantitative  metric 
under  consideration  would  use  the  software  on  each  "field 
unit"  disk  to  record  the  number  of  times  the  different 
Categories  (or  specific  identity)  of  acceptable  practices 
were  reviewed  by  the  "field  users."  A  "qualitative"  metric 
being  considered  is  "user  responses,"  where  users  would 
provide  feedback  about  APMS  and/or  specific  "acceptable 
practices"  (56) . 

Another  improvement  being  planned  by  XRM  is  the 
integration  of  best  practices,  "lessons  learned"  and  other 


55 


"acquisition  references  (such  as  Federal  Acquisition 
Regulations)  like  FARs"  into  the  databases  for  the  AFSC 
Acquisition  Model  and  Database  (AAM&D)  (54:3).  The  AAM&D  is 
the  second  major  acquisition  process  improvement  project 
derived  from  the  APEX  II  team;  it  is  intended  to  be  a  "tool 
to  help  AFSC/AFMC  manage  acquisition  process  improvement  and 
to  aid  program  offices  in  planning/managing  their  programs" 
(54:3).  The  AAM&D  design  will  be  based  primarily  on  the 
recently  completed  Defense  Science  Board  Acquisition 
Streamlining  Task  Force  (DSBTF)  simulation  model,  but  will 
be  altered  to  accommodate  AFSC-specif ic  characteristics  and 
acquisition  task  descriptions  derived  from  the  original  APEX 
effort.  The  eventual  goal  is  to  provide  a  refined  AAM&D 
package  (with  current  databases)  for  users  "in  the  field"  to 
utilize  on  "high-capacity"  personal  computer  (PC)  hardware; 
an  AFSC-based  "center  of  expertise"  would  provide  updated 
AAM&D  packages  and  other  "computer  support"  to  "user" 
agencies  (54:3). 

ASD/CY :  The  purpose  of  ASD/CY  is,  simply  put,  to 

"serve"  as  a  "full  fledged,  functional  office"  for  the 
"program  management,  test  and  evaluation  management, 
configuration  and  data  management,  and  administrative 
support  of  these  functions"  (58:1).  ASD/CY  is  expected  to 
act  "as  the  ASD  focal  point  for  acquisition  management 
policy,"  as  well  as  for  policies  for  "T&E,"  and  "CM/DM,"  and 
as  "home  office  for  acquisition  management  personnel,  both 
military  and  civilian"  (58:5). 
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In  the  "program  management/acquisition  management” 
area,  ASD/CY's  duties  include: 

(a)  Establishment  of  ASD's  Request  for  Proposal 
Preparation  and  Source  Selection  Office  (58:1); 

(b)  "Support  (for)  ASD  organizations  in  obtaining 
acquisition  management  information  in  order  for  them  to 
achieve/ expedite  their  mission  requirements”  (58:8); 

(c)  "Acting  as  focal  point  for  exchange  of  acquisition 
management  related  information  between  ASD  and  other 
government  and  civilian  organizations”  (58:9); 

(d)  "Maintaining  professional  expertise  in  acquisition 
functional  areas  to  provide  advice  to  the  Program  Executive 
Officer,  Designated  Acquisition  Commander,  and  Program 
Directors"  (58:8). 

ASD/CY  is  also  a  member  of  the  AFSC  Program  Management 
Board  of  Directors  -  whose  "underlying  goal"  is  "to 
implement  acquisition  process  improvements  in  a  way  that 
insures  consistency  across  the  command"  (58:10). 

As  ASD's  board  member,  ASD/CY's  responsibilities 
include  (58:10) : 

(a)  "making  available  the  lessons  learned  from  around 
the  defense  acquisition  community"  -  such  as  "program 
management  lessons  learned  and  best  practices"-  to  "ASD  SPOs 
in  a  timely  and  useful  way;" 

(b)  "to  encourage  and  sponsor  improvement  initiatives 
identified  by  people  in  ASD..." 
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In  order  to  accomplish  these  many  duties,  ASD/CY  may 
(at  any  given  time)  need  to  interface  with  HQ  AFSC,  ASD 
SPOs,  other  ASD  functional  offices,  and  possibly  PEOs, 
contractors  and  others. 

The  "justification”  for  this  new  agency  was:  ASD/CY  was 
formed  in  order  to  address  DOD  and  Congressional  concerns 
about  program  manager  support  for  the  PEOs,  and  to  deal  with 
new  requirements  to  trcin  and  provide  experienced 
acquisition  management  personnel  to  support  an  "Acquisition 
Corps"  (59) . 

ASD/CY  has  no  "formal"  lessons  learned  program  of  its 
own;  it  does,  however,  support  the  AFSC  Program  Management 
Lessons  Learned  Program  at  the  ASD  level.  This  is  done 
primarily  through  ASD/CY 's  coordination  of  the  ASD's 
bimonthly  Program  Management  Seminars.  These  seminars  are 
attended  by  the  Program  Directors  (and  their  deputies)  from 
ASD's  various  program  offices;  presentations  are  given  by 
representatives  from  selected  SPOs,  describing  certain 
experiences  and  useful  knowledge  -  such  as  lessons  learned  - 
gained  during  program  cycles  (59).  These  seminars  appear  to 
be  ASD/CY 's  means  of  complying  with  the  "face  to  face 
meetings  between  SPOs"  requirement  of  AFSCR  550-19. 

ASD/CY  also  supports  "information  transfers"  (such  as 
lessons  learned)  at  quarterly  meetings  between  the  "product" 
division  SPOs,  functional  offices,  and  the  PEO  offices  (and 
sometimes  also  HQ  AFSC  agencies) .  Such  meetings  are  usually 
attended  by  "staff  level"  personnel,  and  offer  relatively 
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informal  avenues  for  obtaining  or  providing  "lessons 
learned"  to/ from  the  attendees  (59) . 

ASD/CY  obtains  much  of  its  "functional"  information 
from  HQ  AFSC/XR  -  which  provides  policies  and  guidance  on 
program  management  matters  -  and  HQ  AFSC/DR  (for  test  and 
evaluation  policies)  and  HQ  AFSC/EN  -  for  configuration 
management  policies  (58:2).  Information  about  "local" 
activities  comes  mainly  from  the  ASD  functional  "chiefs," 
the  various  program  directors,  and  the  ASD  Commander;  this 
information  is  often  provided  at  weekly  "staff  meetings" 

(59)  . 

ASD/CY 's  information  "needs"  are  usually  expressed 
informally,  through  telephone  conversations,  personal 
contact,  or  written  memos;  their  "subjects  of  interest"  are 
generally  oriented  by  functional  specialty/program,  or 
special  interests  (59) . 

Information  provided  to  ASD/CY  may  range  from  "summary" 
information  -  such  as  bulletins  provided  via  "electronic 
mail"  -  to  "Technical  Report"  level  -  such  as  written 
"paper"  programming  plans  (59). 

Much  information  is  furnished  to  ASD/CY  using  telephone 
conversations  and  personal  contact  at  meetings,  in  addition 
to  written  and  electronic  means.  The  information  may  be 
maintained  in  a  medium  or  document  type  similar  to  that  in 
which  it  was  furnished  (such  as  paper  reports)  or 
"transferred"  to  another  medium/form  -  such  as  telephone 
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conversations  summarized  on  an  individual's  computer  disk 
file  memos  or  placed  in  personal  memories  (59) . 

Depending  upon  the  specific  "subject  matter"  supported 
by  the  provided  information,  CY  might  make  few  changes  (such 
as  in  "lessons  learned"  furnished  by  other  agencies)  or  many 
comments  (such  as  when  providing  "consulting"  services  to 
program  directors  -  about  whatever  program  management 
matters  are  of  concern  to  the  PDs) .  Any  such  "modified" 
information  can  generally  be  retrieved  using  the  subject 
matter  as  the  index,  when  using  magnetic  disk  directories  - 
paper  files  may  be  indexed  by  functional  "specialty," 
program  or  the  "special  interest"  subject  (59) . 

ASD/CY  disseminates  information  in  different  mediums 
and  forms,  depending  upon  the  type  of  "subject  matter"  and 
the  intended  "receivers;"  the  ASD  Commander  might  receive  a 
written  paper  "summary"  or  "background  paper"  describing  a 
CY  response  to  a  staff  meeting  action  item,  or  a  multi- 
organizational  "meeting"  might  be  conducted  using 
"videoteleconferencing"  and  information  would  be  provided 
via  "personal"  contact  -  or  any  videotapes  kept  as  records 
of  the  meeting  (59).  Most  "receivers"  are  agencies  that 
"work  with"  ASD/CY  on  a  "regular"  basis,  although  obviously 
requests  for  information  from  "higher  level"  individuals  or 
organizations  (such  as  the  Program  Executive  Officers)  would 
be  answered  accordingly.  As  such,  most  ASD/CY  information 
is  normally  "seen"  only  by  USAF  (and  some  contractor) 
agencies  ( 59 ) . 
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Metrics  are  a  concern  in  ASD/CY;  quantitative  types  are 
favored.  For  example,  RFP  processing  schedules  (actual 
compared  to  predicted)  are  used  to  measure  time  savings 
achieved  from  "up  front”  advice  from  CY's  RFP  Support 
Organization  (as  opposed  to  "normal”  RFP  processing  time) 

(59)  . 

Since  it  is  a  new  organization  -  formation  began  in 
early  1990  (59) ,  the  official  "implementation”  plan  was 
approved  by  ASD/CC  in  May  1991  (60)  -  and  is  organized 
according  to  TQ  principles  -  streamlined  management 
structure,  minimum  staff  support,  and  so  forth  (58:3), 

ASD/CY  is  "in  the  midst”  of  adopting  or  considering  many 
changes  to  its  "way  of  doing  business.”  Examples  include: 

(a)  Using  the  Defense  Systems  Management  College 
(DSMC)  Alumni  Association  (Dayton  Chapter)  meetings  as  an 
"alternate”  forum  for  disseminating  (and  obtaining) 
generally  useful  (and  unclassified/nonsensitive)  acquisition 
management  information  (such  as  lessons  learned) .  This 
arrangement  provides  an  informal  setting,  in  which  all 
"levels"  of  personnel  can  attend  and  ask  r-uestions  -  not 
just  executives  (59) .  The  information  is  generally  provided 
in  a  presentation  -  one  scheduled  topic  was  "Experiencing  a 
Defense  Acquisition  Board,  or  DAB"  -  by  an  individual  who 
is  "well  acquainted"  with  the  subject  (such  as  a  PD)  and  can 
answer  questions  or  clarify  details.  Announcements  of  such 
meetings  are  usually  made  in  "periodicals"  such  as  the  base 
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newspaper,  to  encourage  attendance  by  "interested"  personnel 
(61)  ; 

(b)  Becoming  the  ASD  focal  point  for  the  new 
Acquisition  Program  Tracking  System  (APTS) .  The  APTS 
contains  a  series  of  "metrics"  -  developed  as  part  of  AFSC's 
ongoing  efforts  to  improve  the  acquisition  process  -  that 
will  be  used  by  acquisition  program  management  to  measure 
their  progress  (in  certain  cases  it  may  be  used  as  a 
replacement  for  the  Defense  Acquisition  Executive  Summary, 
DAES,  currently  written  by  certain  program  offices)  (59) ; 

(c)  Using  the  "Quick  Mail"  system  +  send  memos, 
messages,  or  even  conduct  "conferences"  -  through  use  of  the 
computer  electronic  "network"  used  at  ASD  (59) ; 

(d)  Developing  new  metrics  -  one  possibility  would  be 
a  measurement  of  "effective  contract (or)  selection  process" 
by  tracking  the  "meantime"  intervals  between  contract  awards 
and  "yellow/red  flag"  progress  reports  -  such  a  metric  could 
be  used  to  help  analyze  the  "up  front"  contributions  of 
various  advisory  groups  such  as  the  RFPSO  (59) ; 

(e)  Investigating  the  various  "reviews  and  audits" 
performed  by  ASD  or  other  agencies  on  SPOs,  to  identify 
areas  where  "up  front"  help/advice/support  could  (perhaps) 
reduce  the  number  of  "inspections"  needed  -  an  example  of  a 
desirable  result  would  be  "eliminating  (Independent  Cost 
Assessments)  ICAs  by  validating  cost  estimating  models  used 
by  SPOs"  (62) . 
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Evaluator  Agencies;  The  agencies  sampled  in  this 


subdivision  include  the  General  Accounting  Office  (GAO) , 
Cincinnati  Regional  Office,  Dayton  site;  and  the  Air  Force 
Inspection  and  Safety  Center,  Systems  Acquisition  Management 
Inspection  Division  (AFISC/IGY) . 

GAO  fPavton  site^ ;  The  purpose  of  the  GAO  is,  simply 
put,  to  investigate  and  evaluate  various  programs,  projects, 
agencies,  and  issues,  as  directed  (or  desired)  by  committees 
and  (sometimes)  individuals  in  the  U.S.  Congress.  The  cadre 
of  GAO  personnel  at  the  Dayton,  Ohio  (Wright-Patterson  AFB) 
suboffice  tend  to  concentrate  on  Congressional  concerns  that 
relate  to  the  development,  acquisition  and  deployment  of 
major  USAF  aircraft  and  related  systems  (63) . 

Given  this  area  of  responsibility,  the  GAO  "cadre"  is 
able  to  (and  usually  does)  work  with  any  agencies  connected 
with  whatever  programs  are  being  investigated.  This 
includes  "user"  commands,  defense  contractors, 
subcontractors,  and  subtier  contractors,  test  &  evaluation 
centers,  and  various  DOD  and  Service  headquarters  agencies, 
major  commands.  Program  Executive  Offices,  laboratories,  and 
SPOs  (64). 

The  GAO  possesses  the  authority  to  task  all  these 
agencies  for  cooperation  with  their  investigations.  This 
authority  is  described  in  GAO  Order  0110.1,  "Legislation 
Relating  to  the  Functions  and  Jurisdiction  of  the  General 
Accounting  Office,"  which  outlines  the  various  pieces  of 
Congressional  legislation  dealing  with  this  subject,  even 
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"all  the  way  back”  to  the  Budget  and  Accounting  Act  of  1921 
which  originally  established  the  GAO  (64). 

The  basic  types  of  information  normally  sought  by  the 
GAO  are  those  that  deal  with  the  financial  and  management 
performance  aspects  of  programs  or  agencies  -  essentially 
"exactly  what  was  your  money  spent  for,  and  how  well  did/ is 
the  program  succeed (ing)  in  providing  the  authorized 
product (s)  and/or  service (s)?"  In  recent  years,  however. 
Congress'  increasing  use  of  the  GAO  as  its  "question 
answering"  and  "legislation  verification"  tool  has  altered 
the  scope  of  the  agency's  interests.  Now,  those  areas  of 
information  that  are  expected  to  answer  the  Congressional 
questions  are  included  in  GAO  interests  -  even  if  the  areas 
are  not  particularly  associated  with  financial  or  management 
issues  (63)  . 

The  GAO  does  not  possess  a  formal  "lessons  learned"  or 
similar  program;  their  reports  to  their  Congressional 
"sponsors"  do  discuss  key  points  and  factors  relating  to  the 
issues  in  question  and  the  actions  they  recommend  to  address 
"sponsor"  concerns.  Most  of  these  reports  are  unclassified 
and  therefore  usually  become  "public  documents"  no  later 
than  30  days  after  delivery  to  the  sponsor  (64). 

When  given  an  investigation  "tasking,"  the  cognizant 
GAO  office  (usually  a  field  office  such  as  the  Cincinnati 
Regional  office,  or  its  Dayton  subunit)  puts  together  an 
"investigation  planning  team."  This  team  develops  a 
detailed  plan  of  operations  for  the  investigation  (63). 
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This  plan  addresses  the  scope  of  the  investigation,  the 
types  of  information  needed,  the  number  of  investigations 
required,  the  time  length  of  the  investigation,  judgement 
criteria,  and  similar  factors.  The  plan  is  studied  and 
modified  (if  needed)  by  a  review  group  and  the  planning  team 
(63)  . 

Once  the  plan  is  approved,  the  cognizant  GAO  office 
notifies  the  primary  "affected”  organization  of  its  intended 
"audit"  schedule  and  the  information/agency  access  required. 
Such  notifications  are  usually  in  a  written  "letter"  form, 
and  are  regarded  as  official  correspondence  (63) . 

The  investigation  team  often  "sits"  within  the  primary 
"investigated"  agency  (at  least  part  of  the  time) ,  and 
travels  to  other  sites/organizations  as  needed.  The  team 
usually  reviews  that  agency’s  management  and  financial 
plans,  status  reports,  and  similar  documents,  as  well  as 
whatever  agency  materials  relate  to  the  "topics  of  interest" 
(63) .  Depending  upon  the  those  specific  topics,  the 
investigators  will  interview  various  personnel  within  the 
affected  agencies,  ranging  from  the  program  director  to 
specific  submanagers,  analysts,  and  functional  specialists. 

A  similar  "range"  of  documents,  presentation  materials,  and 
working  papers  may  also  be  requested,  if  the  investigators 
feel  that  degree  of  detail  is  needed  to  properly  understand 
the  "macro"  implications  of  the  program  activities.  This 
approach  is  quite  feasible  (from  the  investigators' 
viewpoint)  since  the  GAO  team  may  often  spend  several  months 
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"interfacing"  with  the  affected  agencies  at  the  agencies' 
locations  -  where  such  documents  should  be  readily  available 

(63)  . 

When  the  investigators  "write  up"  their  findings,  they 
are  able  to  make  the  appropriate  changes  to  the  structure 
and  content  of  the  obtained  information.  These  changes  may 
be  needed  in  order  to  explain  and  clarify  the  findings  (and 
recommendations)  for  the  sponsor.  However,  the 
investigators  are  expected  to  review  their  finished  product 
"line  by  line"  to  insure  that  every  statement  can  be 
"referenced"  by  actual  documentation  or  interview  records 

(64)  .,  Such  reviews  are  supposed  to  be  so  complete  as  to 
allow  a  "new"  GAO  member  to  clearly  understand  and  "follow" 
the  rationales  used  in  the  report  (64) . 

All  writeups  are  supposed  to  address  the  subject 
categories  found  in  the  investigators'  Findings  Summary 
Sheet,  which  are  shown  as  follow  (65) : 

(a)  Issues  Statement  (reasons  for  the  investigation) ; 

(b)  Criteria  (by  which  the  findings  are  judged) ; 

(c)  Conditions  (discovered  during  the  investigation) ; 

(d)  Cause  (of  the  conditions) ; 

(e)  Effect  (on  project,  other  programs) ; 

(f)  Conclusions  (about  the  situation); 

(g)  Potential  Recommendations  (for  actions  to  be  taken 
by  affected  agencies  and  others  involved) ; 

(h)  Potential  Alternatives  (to  the  recommended 
actions) . 
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The  final  "report"  for  the  sponsor  is  sometimes  given 
in  briefings  or  presentations,  but  is  most  often  in  an 
actual  report  (63).  This  researcher's  review  of  several 
different  GAO  reports  indicated  that  these  reports  contain 
an  introduction  section,  and  sections  for  discussion, 
findings,  conclusions  and  recommendations.  Such  reports 
usually  have  some  form  of  "executive  summary"  which 
"highlights"  the  audit  purpose,  background,  results-in- 
brief,  principal  findings,  any  matters  for  congressional 
consideration,  and  (GAO)  agency  comments.  Most  reports  also 
contain  appendices,  which  often  contain  comments  from  the 
"affected"  organizations,  and  any  references  obtained  from 
other  GAO  documents.  The  principal  investigating  office  and 
the  maj.or  team  members  are  identified  at  the  end  of  each 
report  (66,  67,  68,  69). 

The  reports  are  almost  always  provided  in  paper  form, 
and  are  addressed  to  the  chairmen  of  the  relevant  committees 
or  the  appropriate  sponsor  (63).  The  Publishing  Office  at 
GAO  "Headquarters"  maintains  a  list  of  special 
activities/subjects  that  various  Congresspersons  or 
committees  desire  to  be  kept  informed  about;  if  a  report 
deals  with  any  of  those  interests,  copies  will  be  provided 
accordingly  (64).  Copies  of  any  reports  that  involve  DOD- 
related  issues  are  supposed  to  be  provided  to  the  cognizant 
DOD  agencies,  which  are  then  expected  to  distribute  the 
copies  to  the  appropriate  lower-level  organizations.  The 
"affected"  organizations  often  receive  "courtesy  copies"  of 
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the  document  from  the  investigating  office,  rather  than  the 
'•normal”  source,  the  Publishing  Office  (64)  .  Most  reports 
contain  a  section  which  describes  procedures  for  ordering 
(extra)  copies,  and  states  whatever  fees  are  required. 

The  "investigating”  GAO  agency  maintains  some  copies 
of  their  reports,  plus  (for  at  least  three  years)  the 
various  working  papers  and  other  "reference”  documents 
gathered  by  the  team.  These  "paper"  documents  are  generally 
filed  by  program  category  (B-1  program,  for  example)  (64) . 

If  the  GAO  personnel  wish  to  research  subjects  not  in 
their  "on  hand”  documents  (such  as  prior  to  or  in  support  of 
an  investigation) ,  they  can  consult  published  paper  copies 
of  the  GAO'S  Reports  and  Testimonies  Indexes;  these 
documents  are  published  monthly  or  annually,  and  contain 
report  titles  (both)  and  synopses  (monthly  version  only) 

(64) .  The  personnel  might  also  access  a  GAO  electronic 
"report  directory,"  which  uses  a  "key  word"  search  software 
to  locate  report  listings  by  title  subject (s)  or  report 
numbers  (64) .  The  reports  can  then  be  ordered  from  one  of 
many  National  Repositories  for  public  documents,  such  as  the 
one  located  in  Cincinnati  OH  (64). 

GAO  personnel  also  consult  various  members  of  their  own 
office  when  seeking  information;  many  investigators  have 
been  in  the  same  office  for  several  years  and  have  thus 
accumulated  a  wealth  of  "corporate  memory"  about  various 
programs/ subject  areas.  This  situation  is  particularly  true 
at  the  Dayton  GAO  site;  certain  individuals,  for  example. 
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have  monitored  the  B-IB  program  since  its  inception  in  1981 
(63). 

The  "metrics”  used  by  the  GAO  to  evaluate  their  efforts 
are  not  usually  quantitative  in  nature.  The  central 
headquarters  does  track  the  number  of  investigations,  for 
example,  but  this  information  is  generally  used  only  in  the 
Annual  Reports  to  Congress,  which  deal  with  agency  budget 
considerations,  not  "information  processing"  (64) . 

The  GAO'S  various  qualitative  measures  generally 
address  the  procedures  and  content  of  the  investigation. 
There  are  several  documents,  among  them  the  GAO's 
"Government  Auditing  Standards,"  the  "GAO  General  Policy 
Manual,"  and  the  "GAO  Communications  Manual,"  that  contain 
extensive  descriptions  of  the  standards  and  guidelines 
required  for  GAO  investigations.  These  requirements  are 
intended  to  insure  the  investigations  are  conducted  in  a 
thorough,  accurate,  and  impartial  manner  (70,  71,  72) . 

These  requirements  also  apply  to  the  investigation  reports; 
an  extensive  review  process  is  used  to  insure  the  reports 
"meet  standards"  before  the  report  is  published  or  forwarded 
to  the  sponsor  (64) . 

There  is  "feedback"  from  the  GAO's  "customers"  -  their 
sponsors  -  in  many  cases,  but  such  feedback  is  usually 
informal,  often  general  in  nature,  and  may  sometimes  contain 
inappropriate  comments.  Such  comments  might  come  from  "non¬ 
sponsors"  such  as  opposing  parties  (or  even  the  sponsors 
themselves) ,  and  may  express  dissatisfaction  with  the 
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content  of  the  report  (73) .  It  is  reasonable  to  assume  that 
such  dissatisfaction  is  (in  most  cases)  based  on  political 
considerations  rather  than  the  accuracy  and  scope  of  the 
"information  processing”  efforts. 

One  change  being  implemented  by  the  GAO  (at  least  for 
the  Advanced  Tactical  Fighter  program)  is  the  "placing"  of 
investigating  teams  "periodically"  into  the  program  office, 
staring  at  earlier  stages  in  the  development  life  cycle;  the 
GAO  has  been  involved  with  the  ATF  since  the  beginning  of 
concept  exploration  (73) .  It  is  reasonable  to  assume  that 
any  significant  issues  arising  from  continued  development  of 
the  ATF  will  be  brought  to  the  attention  of  Congress  and  the 
public,  probably  earlier  in  the  system  life  cycle  than  on 
previous  similar  programs. 

AFISC/IGY;  The  purpose  of  AFISC/IGY  is  - 
basically  -  "to  assess  (USAF  acquisition)  process (es)  and 
program  effectiveness,"  to  "identify  successful  and 
deficient  acquisition  management,"  and  "to  consult  and 
advise"  various  USAF  agencies;  the  "tool"  IGY  uses  to 
accomplish  these  actions  is  the  Systems  Acquisition 
Management  Inspection,  or  SAMI  (74). 

The  intent  of  SAMIs  is,  in  IGY's  view,  to  insure  that 
there  is  "value  added"  to  the  process  or  program  under 
review,  rather  than  to  focus  on  the  "compliance  issues"  that 
many  agencies  normally  associate  with  visits  from 
"inspection"  agencies.  Such  added  value  is  provided  by: 
"focusing"  the  SAMI  "on  major  issues;"  proposing  "realistic 
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solutions”  to  any  identified  problems;  preventing  "train 
wrecks”  in  the  program;  closing  the  "credibility  gaps"  found 
in  various  agencies  (including  the  IG) ;  and  providing 
managers  with  an  "unbiased,  independent,  expert  source  for 
second  opinions"  that  is  also  "within  the  (USAF)  family" 

(74)  . 

SAMIs  can  be  used  to  address  issues  in  any  part  of  the 
acquisition  life  cycle  of  a  program,  "from  requirements  to 
supportability . "  Thus,  since  IGY's  "area  of  responsibility" 
is  considered  to  be  "Air  Force-wide,"  that  office  would 
usually  work  with  a  spectrum  of  agencies  -  "users"  such  as 
Strategic  Air  Command  (SAC) ,  the  Air  Staff,  PEOs,  SPOs,  Air 
Logistics  Centers,  and  HQ  AFSC,  among  others  (74) . 

IGY's  basic  authority  to  conduct  SAMIs  stems  from  The 
Inspector  General  (TIG) ,  Office  of  the  Secretary  of  the  Air 
Force  (OSAF) ,  who  is  "who  AFISC  works  for."  The  TIG's 
authority  stems  from  "Title  10  US  Code,  8020",  which  tells 
the  TIG  to  (75) : 

(a)  "Inquire  into  and  report  on  discipline, 
efficiency,  and  economy  of  the  Air  Force;" 

(b)  "Periodically  propose  programs  of  inspection  and 
recommend  additional  inspections  and  investigations;" 

(c)  Perform  "any  other  duties  prescribed  by  the 
(Secretary  of  the  Air  Force)  SECAF  or  (Chief  of  Staff  of  the 
Air  Force)  CSAF." 

Approved  topics  for  these  investigations  are  provided 
to  TIG/AFISC/IGY  by  the  Air  Staff,  which  "prioritizes"  the 
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many  proposed  topics,  so  that  AFISC  can  plan  for  and 
allocate  the  resources  and  personnel  needed  to  conduct  the 
"high  priority"  investigations  (76) .  "Proposed  topics"  may 
come  from  many  sources,  such  as  SECAF  or  CSAF,  HQ  USAF,  SAF 
(the  SECAF' s  support  agencies),  as  well  as  the  Major 
Commands  (SAC,  AFSC,  and  others) ,  the  PEOs,  Office  of  the 
Secretary  of  Defense  (OSD) ,  and  also  individual  program 
offices  or  agencies  (74,  77);  IGY/AFISC  can  also  provide 
"proposed  topics,"  which  must  be  approved  by  TIG  before 
submittal  to  the  Air  Staff  (78) .  In  those  cases  where  IGY 
proposes  topics,  they  often  conduct  one  or  two  "data- 
gathering"  trips  prior  to  the  final  Air  Staff  "selection," 
so  that  the  proposal  can  be  revised  to  reflect  more  current 
and  "detailed"  information  (about  tasks,  resources,  and 
other  issues) .  Once  the  topics  have  been  finalized.  Air 
Staff  appoints  a  focal  point  for  each  topic,  to  address  any 
questions  the  investigating  agencies  might  have  about  that 
topic  (78) . 

The  AFISC/IGY  can  provide  information  about  many  of  its 
SAMIs  to  many  USAF  organizations  outside  the  "TIG  to  SECAF" 
hierarchy.  However,  certain  information  is  considered 
"sensitive"  or  "for  official  use  only,"  so  all  results  must 
be  screened  (and  edited  if  needed)  before  any  "public" 
release  (78) . 

IGY  does  not  have  a  formal  "lessons  learned"  process  or 
program.  Periodically  (approximately  every  two  years) ,  a 
"special  book,"  with  observations  culled  from  many 
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investigations,  may  be  created  -  however,  this  document  is 
usually  limited  in  its  distribution,  since  it  often  contains 
"sensitive"  or  similar  information  (78) . 

Once  IGY  has  received  its  "priority  proposals"  list, 
it  plans  the  various  SAMIs  -  schedules,  agencies  to  be 
visited,  team  "specialty"  composition  and  size,  special 
"subjects  of  interest"  and  similar  factors  are  considered. 

In  addition,  the  planners  often  coordinate  their  actions 
with  the  Acquisition  Oversight  Control  Working  Group 
(AOCWG)-  a  collection  of  representatives  from  various 
"evaluator"  agencies  such  as  AFISC,  the  IGs  from  the  Major 
Commands,  and  the  Air  Force  Audit  Agency  -  to  insure  the 
team  would  not  be  interfering  with  other  agencies' 
investigations.  Then  formal  notice  is  given  to  the  "main" 
agencies  of  interest  -  usually  in  a  formal  paper  letter  -  of 
the  intended  schedule,  so  that  proper  facilities  can  be  made 
available  for  use  by  the  team  (76) . 

Upon  arrival  at  the  "primary"  agency  to  be  visited,  the 
SAMI  team  conducts  an  "in-briefing"  for  agency  management. 
This  briefing  explains  the  reasons,  goals,  and  objectives  of 
the  SAMI,  and  provides  a  forum  for  the  team  to  request 
assistance  and  the  agency  management  to  offer  their  views 
and  comments  concerning  the  "proper  course"  for  the  SAMI 
(79)  . 

Depending  upon  the  topic  of  interest,  a  SAMI  team  may 
obtain  information  from  various  "levels"  in  the  :  .vestigated 
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agencies,  from  the  program  managers/directors  to  specific 
"specialists”  in  the  functional  areas  or  management  offices. 
This  information  tends  to  be  in  a  "level  oriented”  form  - 
"summary"  information  from  higher  levels,  "background"  data 
at  medium  levels,  or  "detailed  report"  information  at  the 
"worker  bee"  levels.  The  information  may  be  provided  in 
paper  documents,  computer  files,  videotapes,  telephone 
conversations,  or  personal  interviews,  or  other  mediums  as 
"appropriate"  -  or  as  outlined  in  various  IGY  office 
procedures  and  instructions  relating  to  the  conducting  of  a 
SAMI  (78)  . 

The  SAMI  team  may  "rearrange"  the  information  they 
receive  into  whatever  form  needed  to  address  the  "topic  of 
interest;"  however,  they  must  reference  the  original  sources 
of  the  information,  and  also  explain  the  context  in  which 
that  information  was  contained  (78)  . 

When  the  SAMI  team  finishes  its  work,  they  present  the 
results  of  the  investigation  (and  their  conclusions  and 
recommendations)  to  the  agency  management  at  an  "out- 
briefing."  This  information  is  usually  contained  in  paper 
copies  of  charts  or  the  actual  "vugraph"  overhead  projection 
transparencies.  However,  the  official  "writeup"  of  their 
work  is  almost  always  put  in  a  paper  "report"  form  (79) . 

This  report  contains  an  "executive  summary"  section,  which 
"highlights"  the  SAMI  purpose,  scope,  background,  findings 
and  recommendations,  observations,  and  posr  inspection 
update  sections;  the  report  also  contains  attachments  to 
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explain  acronyms,  identify  the  visited  organizations, 
identify  the  team  members,  and  other  needed  information.  At 
the  end  of  this  report  is  a  section  with  "reply 
instructions"  -  for  agencies  that  must  take  action (s)  to 
correct  findings  (which  are  considered  problems  or 
deficiencies)  and  a  "distribution  table"  that  identifies: 
the  agencies  that  will  receive  this  report;  the  #  of  copies 
they  will  get;  the  paragraphs  in  the  report  that  explain 
what  kind  of  reply  is  needed;  and  the  type  of  "OPR"  that  the 
reply  must  be  sent  to,  plus  the  suspense  time  interval  for 
the  reply  (79) . 

IGY  maintains  the  information  from  SAMIs  in  several 
different  mediums  and  formats.  The  paper  reports  (and  other 
paper  documents)  are  very  common,  as  are  (of  course)  the 
personal  memories  of  the  SAMI  team  members.  Computer 
magnetic  disks  are  sometimes  used  -  for  working  papers, 
draft  .correspondence,  and  similar  items  (78).  Specific 
findings  and  observations  can  also  be  maintained  in  the 
Inspection  Reporting  Information  System  (IRIS) .  IRIS  is  an 
electronic  database  controlled  by  the  Systems  Development 
and  Followup  Branch,  IGPD  (80) .  These  "findings  and 
observations"  are  "formatted"  into  IRIS  (by  IGPD  personnel) 
using  IGY  (or  other  agencies)  inputs  "placed"  on  the  AFISC 
Form  0-41,  "Inspection  Finding  Worksheet"  (80) ;  filling  out 
the  worksheet  requires  the  use  of  the  accompanying  IRIS 
Coding  Instruction  booklet,  and  the  "code  definition"  lists, 
which  identify  the  numerical  codes  associated  with  various 
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subject  areas,  organizations,  functional  areas,  actions, 
problem  causes,  classes  of  loss,  and  significances  (81,  82) . 

IGY  personnel  maintain  unofficial,  "working”  files  that 
are  generally  indexed  by  whatever  those  individuals'  current 
SAMIS  indicate  are  "subjects  of  interest;"  official  files  - 
that  contain  official  correspondence,  replies  to  SAMI 
findings,  "record  copies"  of  reports,  and  similar  items  - 
are  kept  by  the  Plans,  Policy  and  Systems  Division  (IGP) 
office.  If  IGY  personnel  need  to  find  or  retrieve 
information,  they  can  request  it  from  IGP  (if  they  feel  it 
is  "official"),  ask  other  IGY  members,  search  through  their 
own  work  files,  or  work  with  IGPD  personnel  to  check  the 
IRIS  database  (78) . 

If  IRIS  use  is  desired,  the  searcher  may  request 
retrieval  using  a  large  variety  of  information  "keys,"  that 
are  based  on  the  data  input  onto  the  Form  0-41.  Such  "keys" 
include;  subject  areas,  project  number,  type  of  inspection, 
the  Division  which  performed  the  inspection,  the  OPR  for  the 
finding,  date  of  finding,  functional  areas,  management 
actions,  problem  causes,  and  report  status.  Care  must  be 
taken  with  such  a  search,  to  help  the  would-be  information 
user  "save  time  in  'sorting  through'  computer  output  data" 
(81). 

IRIS  outputs  are  on  "computer  paper,"  and  are  intended 
only  for  use  by  AFISC  personnel.  Paper  reports  are  often 
sent  to  other  Air  Force  agencies,  since  the  restrictions 
generally  apply  only  to  non-USAF  agencies  (78) .  However, 


there  is  one  form  of  "paper”  document  that  can  include  SAMI- 
related  information  and  also  be  released  outside  the  Air 
Force:  the  "TIG  Brief,"  AFRPll-1.  This  periodical  is  used 
to  "spread  the  word"  about  subjects  and  issues  that  the  TIG 
office  feels  would  benefit  Air  Force  members  "if  they  knew 
about  them."  IGY  personnel  can  write  articles  and  submit 
them  for  TIG  review  (and  hopefully  publication) ;  article 
subjects  may  include  SAMI  topics,  as  long  as  the  "sensitive" 
information  has  been  removed  or  "genericized"  to  avoid 
"inappropriate"  comments  about  specific  Air  Force  programs 
or  agencies  (76) .  Some  examples  of  IGY  articles  published 
in  "TIG  Brief"  include:  "Spares  Acquisition  Integrated  with 
Production,"  in  the  November-December  1990  issue;  "Design  of 
Experiments  in  System  Acquisition,"  in  the  July-August  1990 
issue;  and  "Artificial  Intelligence  as  a  TQM  Tool,"  in  the 
May-June  1990  issue  (83,  84,  85). 

IGY  does  not  appear  to  use  many  "quantitative"  metrics 
to  evaluate  their  "information  transfer"  efforts  (although 
the  number  of  SAMIs  actually  performed  is  probably  tracked 
for  budgetary  planning) ;  IGP  tracks  the  number  of  replies  to 
SAMIs  (and  the  due  dates) ,  and  the  distribution  of  "mailed" 
report  copies  (78)  .  Most  "qualitative"  metrics  involve 
informal  (and  formal)  feedback  from  inspected  agencies, 
"interested  bystanders"  such  as  support  agencies,  and  TIG 
Brief  readers;  if  the  comments  refer  to  "content"  of  a 
SAMI , IGY  is  given  the  comments  and  tasked  to  respond  - 
otherwise,  IGP  handles  the  response.  The  apparent  "lack  of 
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metrics'*  is  causing  concern  in  IGY,  as  other  "evaluator" 
agencies  such  as  the  AFSC  IG  are  apparently  "moving"  to 
adopt  detailed  metrics  (78) . 

IGY  has  been  considering  many  changes,  and  is  working 
to  adopt  several  of  them.  For  example,  surveys  (for 
organizations  such  as  investigated  agencies)  are  being 
discussed,  to  measure  agency  satisfaction  with  such  SAMI 
actions  as  "in-briefs"  and  "out-briefs"  and  possibly  other 
procedures;  coordination  with  other  "evaluator"  agencies  is 
being  encouraged,  such  as  expanded  work  with  the  AOCWG  and 
the  proposed  "linking"  of  the  IRIS  database  to  the  AFAA 
electronic  database,  which  (like  IRIS)  is  also  located  at 
Norton  AFB  (where  part  of  the  AFAA  "resides") ,  but  is  in 
another  building  (78) .  Another  idea  that  "has  been 
mentioned"  is  to  publish  certain  IGY  articles  in  "non  IG- 
oriented"  periodicals,  to  reach  different  "readerships"  and 
to  profile  their  acquisition/management  orientation  (76) . 

Oversiaht/Manaqement  Agencies:  The  agencies  in  this 
sample  subdivision  include  the  Air  Force  Program  Executive 
Office  for  Tactical  Strike  (AFPEO/TS) ,  and  the  Air  Force 
Program  Executive  Office  for  Tactical/Airlift  (AFPEO/TA) . 

AFPEO/TS ;  The  Program  Executive  Officer  (PEO) 
for  Tactical  Strike  (TS)  is  responsible  for  a  set  of 
major/selected  acquisition  programs  that  are  being  done  to 
meet  USAF  (and  joint  service)  "operational  user" 
requirements  for  tactical,  "air-launched"  weapons  and  "air- 
directed  ground  support"  capabilities.  He  (like  all  Air 
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Force  PEOs)  is  directly  accountable  to  the  Assistant 
Secretary  of  the  Air  Force  for  Acquisition  (who  is  also 
referred  to  as  the  Air  Force  Acquisition  Executive,  AFAE,  or 
the  Air  Force's  Service  Acquisition  Executive,  SAE)  and  is 
expected  to  ensure  the  successful  operation  -  in  matters 
such  as  "cost,  schedule,  and  performance  (within  baseline)" 
of  those  programs  (86) .  The  TS  PEO  (as  well  as  any  of  one 
of  his  fellow  PEOs)  is  able  to  "exercise  his  authority"  in 
several  ways,  such  as  "issuing  program  direction  to  the 
program  directors,  baselining  each  program  according  to  the 
(Acquisition  Program  Baseline)  APB  process,  and  having  the 
(program  directors)  PDs  report  directly  to  him"  (86) .  The 
programs  within  the  TS  "portfolio"  include:  the  Advanced 
Medium  Range  Air-to-Air  Missile  (AMRAAM) ;  the  Sensor  Fuzed 
Weapon  (SFW) ;  the  Joint  Surveillance  and  Target  Attack 
System  (JSTARS) ;  the  Tacit  Rainbow  antiradiation  cruise 
missile;  the  Direct  Airfield  Attack  Combined  Munition 
(DAACM) ;  and  just  recently  added,  the  AGM-130A  "boosted 
glide"  guided  bomb  (87,  88). 

Given  these  responsibilities,  the  AFPEO/TS  personnel 
are  able  to  (and  usually  do)  work  with  a  wide  range  of  Air 
Force,  other  service  and  industry  organizations.  The  most 
frequent  interfaces  are  with  the  various  SPOs,  and  their 
Program  Element  Monitors,  or  PEMs  (who  are  concerned  with 
matters  pertaining  to  the  planning,  programming  and 
budgeting  process),  who  sit  in  the  AFAE's  "staff"  office, 
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SAF/AQ  (89) .  TS  personnel  also  have  extensive  contacts  with 
(88)  : 

(a)  "User”  agencies,  such  as  Headquarters  Tactical  Air 
Comiaand  (HQ  TAG)  ; 

(b)  the  Army  and  Navy  focal  points  for  joint 
developments  (like  AMRAAM  and  JSTARS) ; 

(c)  Resource-providing  agencies  such  as  HQ  AFSC,  and 
the  Aeronautical  Systems  Division  (ASD)  and  the  Electronic 
Systems  Division  (ESD) ; 

(d)  Supporting  agencies  such  as  the  Acquisition 
Logistics  Division  (which  "oversees"  the  Air  Logistics 
Centers) ; 

(e)  Independent  testing  agencies  such  as  the  Air  Force 
Operational  Test  &  Evaluation  Center  (AFOTEC) ; 

(f)  Contractor  corporate  and  program  management  offices 
for  all  of  the  relevant  programs. 

The  authority  given  to  the  PEO/TS,  to  perform  his 
mission  and  interface  with  such  disparate  groups,  stems  from 
the  AFAE's  12  November  1990  "Charter  for  Air  Force 
Acquisition  Program  Executive  Officers  and  Designated 
Acquisition  Commanders"  memorandum  (90) .  That  memorandum  is 
one  form  of  implementation  of  certain  recommendations  from 
the  1986  Packard  Commission's  report  on  the  DOD's  weapons 
accjuisition  system  (11)  (see  Appendix  B  for  a  diagram)  . 

In  keeping  with  those  recommendations,  the  PEO's 
support  staff  has  been  deliberately  limited  in  size  by  the 
AFAE  (8) .  However,  the  PEO  can  organize  his  staff  .into 
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whatever  "division  of  responsibilities"  structure  he  feels 
would  be  most  effective.  In  the  AFPEO/TS  office,  the 
structure  is  primarily  "program-oriented"-  there  are  "action 
officers"  who  are  responsible  for  all  matters  pertaining  to 
specific  programs,  and  also  one  person  responsible  for 
financial  management  issues  of  all  the  programs  (89)  .  There 
is  also  an  "OSD  Liason"  individual,  and  administrative 
support  personnel  (91) . 

The  "program"  structure  has  proven  most  effective  for 
this  TS  office,  particularly  with  regards  to  the 
"firefighting"  and  communications  duties  that  constitute  a 
major  portion  of  the  AFPEO  workload.  This  type  of  workload 
results  from  the  basic  focus  of  AFPEO  personnel:  to  concern 
themselves  with  the  major  problems  and  issues  facing  their 
portfolio  programs  at  any  given  time  -  those  concerns  that 
the  PEO  has  the  authority  and  resources  to  address,  wherein 
an  individual  SPO  might  not  (89) .  An  example  of  such  a 
problem  would  be  finding  funds  to  address  a  contractor's 
unanticipated  (but  justified)  request  for  equitable 
adjustment  to  the  price  of  a  contract  -  the  PEO  has  the 
authority  to  shift  a  certain  amount  of  resources  within  his 
portfolio  from  one  program  to  another,  and  could  (depending 
upon  the  amount  of  the  request)  possibly  make  available  the 
funds  needed  for  that  action  (88) .  Addressing  the  many 
types  of  similar  issues  that  arise,  most  of  them  requiring 
"quick  and  correct"  solutions,  is  commonly  known  as 
"firefighting" . 
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In  order  to  perform  successful  "f iref ights, '•  the 
AFPEO/TS  personnel  concentrate  on  creating  a  communications 
"network"  of  contacts  at  the  various  portfolio-related 
agencies;  this  network  is  used  by  the  TS  staff  to  provide 
and  receive  the  extensive  knowledge  and  cooperation  these 
organizations  need  to  address  "firefighting"  issues  (88) . 

However,  the  TS  "network"  tends  to  contain  only  those 
agencies  and  individuals  that  are  "currently"  involved  (in 
some  manner)  with  the  TS  portfolio  programs.  There  is 
little  communication  with  other  PEO  offices,  except  when 
there  are  significant  "cross-portfolio"  program-related 
issues  -  such  as. past  problems  concerning  AMRAAM  integration 
with  the  F-15  aircraft,  where  the  TS  office  worked  with  the 
AFPEO  for  Tactical/Airlift  (AFPEO/TA)  to  address  the  issues 
(89)  .  Simile-rly,  there  is  little  "general  concerns"  contact 
with  other  non-PEO  agencies;  some  AFPEO/TS  action  officers 
did  attend  the  "short-lived"  Project  Officers  Group 
meetings,  which  were  sponsored  by  HQ  AFSC/XRM  to  address 
such  concerns/ issues  (88) . 

The  AFPEO/TS  office  has  no  structured  "lessons  learned" 
program;  most  of  its  "lessons"  are  provided  informally  - 
"from  memories"  of  TS  personnel  -  to  those  who  seek  the 
information,  such  as  "network"  contacts  or  outside  agencies 
such  as  the  various  IGs  (88) .  However,  the  TS  PEO,  Major 
General  Stephen  McElroy,  has  tasked  the  Tacit  Rainbow  Joint 
Program  Office  (JPO)  to  develop  various  "Lessons  Learned" 
reports,  to  insure  the  "wealth  of  knowledge"  obtained  during 
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that  program's  ten  year  life  is  not  completely  lost  due  to 
its  cancellation  by  the  Secretary  of  Defense  (89) .  One  such 
report,  which  was  circulated  throughout  the  Aeronautical 
Systems  Division  (as  well  as  the  TS  office)  concerned  the 
"lessons  learned"  from  the  source  selection  for  Tacit 
Rainbow's  Mission  Planning  System  (92,  93). 

When  the  TS  office  uses  its  "network"  to  obtain 
information,  they  communicate  with  various  levels  of 
management  within  the  "source"  agencies.  A  PEO  normally 
communicates  with  individuals  "equivalent"  to  him,  in  terms 
of  their  positions  within  their  agencies.  For  example,  the 
PEO  will  deal  with  the  Program  Directors  in  his  SPOs,  and 
usually  the  Chief  Executive  Officers  (CEOs)  of  the  companies 
contracting  with  those  SPOs.  The  AFPEO  action  officers  will 
regularly  seek  information  from  individuals  lower  in  the 
"hierarchy,"  such  as  SPO  projects  directorate  chiefs, 
company  vice  presidents,  and  the  managers  of  the  programs 
being  performed  for  the  SPOs  (88) .  When  dealing  with 
SAF/AQ,  the  PEO  will  normally  work  with  the  Mission  Area 
Directors  (the  individuals  in  charge  of  planning, 
programming,  and  budgeting  functions  for  USAF  acquisition 
efforts  -  by  mission  areas) ,  while  the  action  officers  will 
normally  deal  with  the  PEMs  (who  work  for  the  MA  Directors) 
(88)  . 

The  TS  office  primarily  uses  telephone  calls  to 
describe  its  information  needs  -  verbally  or  through 
"telefaxed"  written  memos  or  letters.  Personal  contact  is 
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also  utilized  extensively;  AFPEO/TS  and  SAF/AQ  are  both 
located  in  the  Pentagon,  while  the  PEO  and  the  action 
officers  travel  extensively,  to  visit  with  various 
government  and  contractor  organizations  within  the  portfolio 
(89)  . 

The  "extent"  of  the  information  obtained  is  dependent 
upon  the  particular  "subjects  of  interest"  at  that  time;  the 
TS  PEO  often  obtains  "executive  summary"  level  information 
(and  the  ’’background"  information  that  supports  the 
summary) ,  while  the  TS  action  officers  usually  obtain 
"status"  information  or  "worker-level"  information,  to  track 
the  program  efforts  at  a  "detailed,"  sometimes  "day  to  day" 
operational  level  (88) . 

The  AFPEO/TS  office  generally  gets  its  information 
through  telephoned  conversations  or  "faxed"  memos  and 
reports,  or  is  given  it  verbally  or  visually  at  various 
meetings  and  briefings.  Videotaped  presentations,  computer 
disks,  "hardcopies"  of  briefing  charts,  or  actual  written 
reports  are  either  "handed  over"  to  the  PEO/action  officers 
at  the  meetings,  or  mailed  "to  the  office"  for  AFPEO/TS  use 
(89)  . 

The  TS  office  can  "modify"  this  information  in  many 
different  ways,  depending  upon  the  "final  use"  to  which  the 
information  shall  be  put.  For  example,  if  the  information 
is  used  in  an  Acquisition  Executive  Monthly  Report  (AEMR) 
for  the  AFAE,  the  useful  data  is  summarized  into  the 
relatively  short,  "executive  summary"  form  desired  for 
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AEMRs.  These  reports  are  also  used  by  the  PEO  to  provide 
his  personal  assessments  of  the  progress  of  his  portfolio, 
and  outstanding  or  upcoming  issues  facing  his  programs  (88) . 
If  a  Defense  Acquisition  Executive  Summary  (DAES)  report  is 
needed  for  the  Undersecretary  of  Defense  for  Acquisition 
(also  called  the  Defense  Acquisition  Executive) ,  additional 
detailed  information  can  be  provided  (since  the  DAES  is 
required  quarterly  rather  than  monthly) .  If  the  data  in 
draft  briefings  is  intended  for  the  Air  Force  Systems 
Acquisition  Review  Council  (AFSARC)  or  the  Defense 
Acquisition  Board  (DAB) ,  the  TS  PEO  visits  the  SPO  and  works 
with  the  PDs  to  develop  the  briefings  (88) .  The  PEO  and 
action  officers  will  review  and  comment  on  SPO-drafted 
documents  such  as  Test  and  Evaluation  Master  Plan  (TEMP) . 
Much  "realtime”  information  is  also  placed  in  "personal 
memos"  or  "Things  to  Do"  lists  (88) . 

The  TS  office  maintains  much  of  this  "modified" 
information  in  an  official  filing  system.  "Paper"  copies 
are  kept  in  folders  in  safes  (or  filing  cabinets) ,  according 
to  specific  file  plans  which  take  into  account  the  different 
programs  and  subjects  of  interest  (88) .  Videotapes  are 
secured  and  arranged  by  program/subject,  while  information 
stored  on  magnetic  disks  is  also  arranged  by  program  or 
subject.  Current,  signed  correspondence  is  kept  in  "read" 
files  (88) . 

Additional  information  is  maintained  by  the  action 
officers  in  personal  "working  folders"  (which  usually 
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contain  working  papers,  draft  documents,  and  whatever 
"current”  information  the  action  officer  is  using) ,  and  (of 
course)  their  personal  memories  (89) . 

The  "modified"  information  is  usually  kept  in  the 
"format"  for  which  it  is  used  ("summary"  for  AEMR, 

"extensive  background"  for  DAES,  and  so  forth) .  If  it  needs 
to  be  retrieved,  the  file  plan  can  be  used  to  find  paper  and 
video  copies,  while  disc  directories  are  used  for  electronic 
media  (88) . 

The  "modified"  information  is  usually  disseminated 
through  mediums  similar  to  those  used  to  obtain  it: 
telephone  conversations  or  "telefaxed"  documents,  mailed 
documents,  or  personally  provided  at  meetings  (89) . 

In  deciding  "who"  should  receive  such  "modified" 
information,  the  TS  personnel  use  mission  requirements,  laws 
or  regulations,  and  their  own  judgement.  "On-file" 
distribution  lists  can  be  used  for  certain  information  (such 
as  the  Tacit  Rainbow  "lessons  learned"  reports) .  Written 
and  verbal  requests  are  also  considered,  depending  upon  the 
identity  of  the  requester,  the  nature  of  the  material,  and 
the  uses  to  which  the  requester  intends  (or  could  intend)  to 
put  such  information.  Such  considerations  are  important, 
particularly  in  situations  where  competing  agencies  may  have 
provided  information  "off  the  record"  or  "in  confidence"  or 
where  legal  or  contractual  ramifications  occur  (88) . 
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Once  such  considerations  have  been  done,  information  is 
"passed  on,"  usually  to  the  appropriate  "network"  member (s) 
or  other  agencies  who  "should"  get  it  (88)  . 

The  AFPEO/TS  office  does  not  use  formal  tvpes  of 
"quantitative"  or  "qualitative"  criteria  to  judge  the 
effectiveness  of  their  "information  processing."  Informally, 
the  TS  personnel  know  who  "provides,"  receives,  and  requests 
the  information;  they  evaluate  any  comments/ feedback 
j-^ceived  and  respond  if  appropriate  (88)  .  For  example,  if  a 
PEM  asks  for  additional  or  clearer  cost  estimates,  TS 
personnel  would  usually  attempt  to  provide  that  information. 
Any  solicitation  of  feedback  is  usually  informal,  such  as  a 
"what  do  you  think  of  this  draft?"  request  to  a  SPO  (88) . 

The  AFPEO/TS  office  has  made  several  recent  changes  to 
its  "infoirmatian  processing."  For  example,  several  of  the 
"network"  contacts  can  exchange  information  using  an 
"electronic"  telephone  "link"  between  computer  terminals. 
This  system  has  transmitted  information  normally  stored  on 
magnetic  disks;  it  has  been  very  useful  for  "real  time" 
information  such  as  briefing  charts  and  status  reports  (89). 
One  action  officer  has  also  placed  himself  on  the  "mailing 
list"  for  electronic  mail  that  the  ASD's  Eglin  AFB  lc'.,cition 
and  ESD  normally  send  to  his  programs  at  those  sites  (SG) . 

The  TS  office  also  has  more  improvements  "in  the 
works."  One  example  is  an  "offsite"  meeting  between  the  PEO 
and  all  of  his  PDs,  currently  planned  for  Fall  1991.  This 
"offsite"  will  offer  an  informal  opportunity  for  the  PDs  to 
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give  comments,  feedback,  and  suggestions  to  the  PEO  (and 
"vice  versa”)  about  how  to  improve  "what  we're  (the 
AFPEO/TS)  doing"  (88) .  In  addition,  in  a  bid  to  become  an 
"information  hub,"  the  TS  office  is  evaluating  a  proposal 
that  would  allow  a  contractor  to  develop  (and  perhaps 
operate  part  of)  an  information  management  system  that  would 
augment  the  TS  communications  "network."  Such  a  system 
offers  the  opportunity  for  the  action  officers  to  "quit 
fixing  the  computers  and  phones"  and  concentrate  on  their 
primary  duties  (88) . 

AFPEO/TA:  The  Program  Executive  Officer  (PEO)  for 

Tactical  and  Airlift  (TA)  is  responsible  for  a  set  of  major 
selected  programs  that  are  being  done  to  satisfy  USAF  (and 
joint  service)  "operational  user"  requirements  for  tactical 
"counter  air"  and  "interdiction"  aircraft,  tactical  "heavy 
airlift"  aircraft,  "tactical  training"  aircraft,  and 
tactical  "friend  or  foe  identification"  capabilities.  He 
has  the  same  basic  responsibilities  to  the  Air  Force 
Acquisition  Executive  (AFAE)  as  those  of  the  PEO  for 
Tactical  Strike  (and  the  other  PEOs) ;  he  is  able  to  exercise 
his  authority  in  the  same  manner  as  his  counterparts.  The 
programs  within  the  TA  "portfolio"  are:  the  F-15  "Eagle" 
air  superiority  and  ground  interdiction  fighter;  the  F-16 
"Fighting  Falcon"  counterair  and  multirole  fighter;  the  F-22 
Advanced  Tactical  Fighter;  the  C-17  advanced  airlifter;  the 
T-IA  Training  System;  and  the  Mark  XV  "Identification  Friend 
or  Foe"  combat  identification  system  (94) . 
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Given  these  responsibilities,  the  AFPEO/TA  personnel 
are  (like  their  AFPEO/TS  "cousins”)  able  to  work  with  a 
spectrum  of  USAF,  "sister"  service,  and  industry  agencies. 
These  personnel  (also)  interface  most  frequently  with  their 
different  SPOs,  the  appropriate  PEMs  in  SAF/AQ.  AFPEO/TA 
personnel  also  have  extensive  contacts  with  (95) : 

(a)  "user"  agencies  such  as  Headquarters  Military 
Airlift  Command  (HQ  MAC),  for  the  C-17; 

(b)  The  Army  and  Navy  focal  points  for  "joint  interest" 
programs  such  as  C-17  and  Mark  XV  (Army)  and  ATF  (Navy) ; 

(c)  resource  providing  agencies  such  as  HQ  AFSC  and 
Aeronautical  Systems  Division  (ASD) ; 

(d)  Supporting  agencies  such  as  the  Air  Logistics 
Centers  (ALCs) ; 

(e)  Independent  testing  agencies  such  as  AFOTEC; 

(f)  Contractor  corporate  and  program  management  offices 
for  all  of  the  "portfolio"  programs. 

The  authority  given  to  the  PEO/TA  is  (as  with  the 
PEO/TS)  based  upon  the  November  1990  "PEO  charter"  written 
by  the  AFAE  (90) .  The  AFPEO/TA  office  is,  like  its 
counterparts,  deliberately  limited  in  size;  the  TA  PEO,  just 
as  other  PEOs,  has  "autonomy"  in  the  formation  of  his  "staff 
structure"  (95) . 

The  AFPEO/TA  office  (in  June  1991)  had  an  "action 
officer"  structure,  where  the  "action  officers"  were 
oriented  to  "mission"  functional  areas.  There  were  four 
action  officers,  one  each  for  Fighter  Programs,  Airlift  and 
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Training  Programs,  Tactical  Systems,  and  Financial 
Management.  There  were  also  executive  and  administrative 
support  personnel  (95,  91) . 

The  AFPEO/TA  personnel  were  also  heavily  involved  in 
"communications”  duties,  in  order  to  "provide  clear, 
unambiguous  direction  to  the  SPOs"  regarding  the  execution 
of  their  programs,  and  to  "provide  the  environment  needed  by 
the  SPOs"  in  order  to  achieve  that  program  execution  (96) . 

The  "firefighting"  activities  similar  to  those 
performed  by  AFPEO/TS,  were  called  "part  of  the  job"  by  TA 
personnel,  although  the  difference  in  terminology  does  not 
reflect  any  difference  in  commitment  (to  resolving  the 
issues  in  question)  (96) . 

The  TA  office  (also)  did  not  have  extensive  dealings 
with  other  PEO  offices  (95) .  However,  when  "cross 
portfolio"  issues  arose,  or  coordination  on  "PEO"  responses 
to  draft  policies  (provided  by  SAF,  for  example)  was 
required,  TA  personnel  would  contact  their  counterparts  in 
other  AFPEOs,  as  needed  (96)  . 

The  AFPEO/TA  office  had  no  formal  "lessons  learned" 
program  or  process.  Certain  formal  "lessons  learned"  were 
acquired,  in  cases  where  special  teams  -  such  as  Program 
Management  Assistance  Teams  (PMATs)  -  were  formed  to  assist 
the  TA  PEO  in  resolving  specific  questions  or  issues. 
However,  those  PMAT  reports  were  usually  considered 
"sensitive"  information  -  often  because  certain  information 
was  provided  "in  confidence"  to  team  members  investigating 
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the  various  agencies  involved  in  the  issues  -  and  therefore 
the  "lessons"  contained  in  the  reports  could  not  be 
disseminated  to  "non-involved"  agencies  (95) .  Most 
"lessons"  were  provided  informally,  often  as  "past 
experiences"  (95,  96) .  Some  "lessons"  that  were  learned  by 
TA  personnel  were  used  as  the  basis  for  their  inputs  to 
the  draft  policies  and  regulations  that  the  AFPPEO/TA  office 
reviewed  -  a  "formal  yet  indirect"  method  of  "passing  on" 
lessons  learned  (96) . 

AFPEO/TA  personnel  also  communicated  with  their 
"contacts"  in  other  agencies  according  to  the  position 
levels  (PEO  to  PD  or  CEO,  action  officer  to  directorate 
chief  or  vice  president)  as  did  TS  personnel  (95,  97) . 

Telephone . conversations,  "telefaxed"  written  memos,  and 
personal  contacts  at  meetings  were  AFPEO/TA  personnel's  most 
preferred  methods  of  requesting,  receiving,  and  providing 
information,  although  formal  written  letters  and  reports 
sent  "through  the  mail"  were  commonly  used  techniques  (95) . 

TA  personnel  (like  TS  personnel)  would  obtain  a  range 
of  "extent"  of  information,  from  "summaries"  for  PEOs  to 
"technical  reports"  for  action  officers.  They  would  also 
(like  the  TS  personnel)  "modify"  received  information  as 
needed  to  provide  correct,  clear  information;  examples 
include  "writeups"  such  as  AFAE  "summary"  reports,  and 
"briefings"  describing  the  AFPEO(TA)  roles  and 
responsibilities  (98,  86,  94). 
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The  TA  office  maintained  much  of  the  "modified" 
information  in  their  official  filing  system.  As  with  the  TS 
office,  paper  copies  were  kept  in  folders  in  cabinets  or 
safes  based  on  specific  file  plans  that  accounted  for  the 
different  "portfolio"  programs  and  subjects  of  interest. 

The  action  officers  also  maintained  informal  "work  folders," 
which  had  magnetic  disks,  working  papers,  and  other  "current 
projects,"  and  also  made  extensive  use  of  their  memories 
(95). 

The  "modified"  information  was  usually  kept  in  the 
"format"  for  which  it  was  used  ("summary"  for  program 
baseline  status  sheets,  "background"  for  management  point 
papers,  "extensive  background"  for  AFPEO/TA  briefings,  and 
so  forth)  (99,  100,  94).  If  it  needed  to  be  retrieved,  the 
file  plan  could  be  used  to  find  paper  copies,  while  disc 
directories  were  used  for  magnetic  media  -  or  other  TA 
personnel  could  be  consulted,  if  the  personal  folders  did 
not  contain  the  information  "either"  (95) . 

The  TA  office's  "modified"  information  was  usually 
disseminated  through  mediums  similar  to  those  used  to  obtain 
it:  telephone  conversations  or  "telefaxed"  documents,  mailed 
documents,  or  personally  provided  at  meetings  (95) . 

In  deciding  "who"  should  have  received  such  "modified" 
information,  the  AFPEO/TA  personnel  used  mission 
requirements,  laws  or  regulations,  and  their  own  judgement 
(like  their  TS  counterparts).  TA  individuals  would  always 
consider  written  and  verbal  requests  based  .upon  the  identity 
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of  the  requester,  the  nature  of  the  material,  and  the  uses 
to  which  the  requester  intended  (or  could  intend)  to  put 
such  information.  Such  considerations  were  important, 
particularly  in  situations  where  competing  agencies  may  have 
provided  information  "off  the  record"  or  "in  confidence" 
(such  as  in  the  "PMAT"  investigations)  or  where  legal  or 
contractual  ramifications  could  occur  (95) . 

Once  such  considerations  had  been  performed, 
information  was  "passed  on,"  usually  to  the  appropriate 
contacts  or  other  agencies  who  the  TA  office  felt  needed  it 
(95)  . 

The  AFPEO/TA  Office  (like  the  AFPEO/TS  office)  did  not 
use  formal  types  of  "quantitative"  or  "qualitative"  criteria 
to  judge  the  effectiveness  of  its  "information  processing;" 
informally,  the  TA  personnel  knew  who  "provided,"  received, 
and  requested  the  information;  they  evaluated  any  comments 
or  feedback  received  and  responded  if  "appropriate."  For 
example,  if  a  PEM  asked  for  additional  or  clearer  cost 
estimates,  TA  personnel  would  usually  attempt  to  provide 
that  information.  Any  solicitation  of  feedback  was  usually 
informal,  such  as  a  "what  do  you  think  of  this  draft?" 
request  to  a  SPO  (95) . 

The  AFPEO/TA  office  is  currently  in  the  midst  of 
substantial  changes,  that  may  or  may  not  significantly 
affect  their  "information  processing"  or  "lessons  learned" 
efforts.  The  former  PEO,  Major  General  Edward  Barry,  has 
been  selected  for  promotion  and  has  moved  on  to  another 
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position.  The  AFPEO/TA  organization  has  been  moved  from 
Wright-Patterson  AFB  in  Ohio  to  the  Pentagon  in  Washington 
D.C.,  to  join  the  other  AFPEO  offices;  only  one  of  the 
action  officers  "accompanied”  the  organization.  The  new  PEO 
will  choose  his  own  "effective"  staff  structure  and 
personnel,  and  thus  determine  the  AFPEO/TA  office's  "ways  of 
doing  business"  (95,  96) . 

Analysis 

Once  all  the  primary  sample  agencies  had  been  surveyed 
(and  the  lack  of  need  to  add  new  samples  to  the 
investigation  was  confirmed) ,  the  "survey"  information  was 
analyzed.  The  intent  of  the  analysis  was  to  identify 
"patterns"  in  the  characteristics  of  the  agencies' 
operations  (that  dealt  with  "information  processing"  or’ 
"lessons  learned") .  The  research  questions  and  the  topic 
checklist  were  used  as  the  bases  for  "pattern  recognition" 
criteria.  The  information  was  nominally  sorted  based  upon 
these  criteria  -  an  agency  "possessed"  the  characteristics 
called  for  in  the  criteria,  or  it  "did  not  possess"  those 
characteristics . 

The  "pattern  recognition"  criteria  covered  six  basic 
categories  of  interest: 

(a)  The  agencies'  abilities  to  obtain  RD&A  project 
management-specific  information; 

(b)  The  type  of  "lessons  learned"  processes  possessed 
by  the  agencies; 
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(c)  The  "modif ications  to  obtained  RD&A  information” 
created  by  the  agencies; 

(d)  The  "maintaining"  of  modified  RD&A  information; 

(e)  The  agencies'  distribution  of  modified  RD&A 
information; 

(f)  The  agencies'  measurement  of  the  effectiveness  of 
their  "information  processing"  or  "lessons  learned"  efforts. 

The  specific  criteria  used  were: 

(a)  Access  to  RD&A  program  management  information  - 
does  the  agency  have  it; 

(b)  Obtaining  RD&A  project  management  information  - 
does  the  agency  actually  do  it; 

(c)  "Structured  Lessons  Learned-Type  Efforts"  -  does 
the  agency,  in  some  manner,  obtain,  maintain,  and  disperse 
various  "lessons  learned-types  of  information"  (or  arranges 
for  other  agencies  to  do  it) ; 

(d)  Formal  Lessons  Learned  Process  -  does  the  agency 
specifically  describe  and  control  "lessons  learned" 
definitions,  formats,  and  acquisition,  storage,  and 
dissemination  procedures; 

(e)  Modification  of  information  -  does  the  agency 
alter  obtained  RD&A  project  management  information  (add 
comments,  emphasize  key  points,  improve  data  clarity,  or 
perform  similar  actions) ; 

(f)  Maintaining  of  information  -  does  the  agency  store 
RD&A  project  management  information  (in  particular 
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'•modified”  information)  such  that  it  can  be  identified  and 
retrieved  (in  a  useful  form)  for  later  review; 

(g)  Distribution  of  information  -  does  the  agency 
routinely  distribute  (or  make  available)  "modified”  RD&A 
project  management  information  to  organizations  not  in  its 
"hierarchy  of  command;” 

(h)  Availability  of  Information  to  "Decision  Makers"  - 
does  the  agency  routinely  distribute  (or  make  available) 
"modified"  RD&A  project  management  information  to 
"oversight,"  "provider"  or  "evaluator"  agencies  not  in  its 
"hierarchy  of  command;" 

(i)  Availability  of  Information  to  "Decision 
Implementers"  -  does  the  agency  routinely  provide  modified 
RD&A  project  management  information  to  "information 
gatherers,"  training  agencies,  or  program  "executers"  (such 
as  SPOs)  that  are  not  in  its  "hierarchy  of  command;" 

(j)  Availability  of  Information  to  "Staff"  -  does  the 
agency  make  the  modified  RD&A  project  management  information 
available  to  personnel  at  "executive"  and  "support"  levels 
within  the  "receiving"  organizations; 

(k)  Metrics  for  Information  Processing  -  does  the 
.agency  (attempt  to)  measure  the  effectiveness  of  its 

"information  processing"  (or  "lessons  learned")  efforts?; 

(l)  Quantitative  Metrics  -  does  the  agency  use 
"recorded"  quantitative  measurements  to  evaluate  the 
effectiveness  of  its  "information  processing"  (or  "lessons 
learned")  efforts?; 
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(m)  Qualitative  Metrics  -  does  the  agency  use 
"recorded”  qualitative  measurements  to  evaluate  the 
effectiveness  of  its  "information  processing”  (or  "lessons 
learned”)  efforts?; 

(n)  Formal  Feedback  -  does  the  agency  obtain  (or 
create)  documented  comments  and  feedback  from  information 
processing/IP  (or  "lessons  learned/LL")  "customer” 
organizations,  concerning  the  agency's  "IP”  (or  ”LL”) 
efforts? ; 

(o)  Solicited  Feedback  -  does  the  agency  use  formal 
techniques  (such  as  questionnaires,  comment  forms, 
interviews,  or  other  "structured"  methods)  to  obtain 
comments/ feedback  from  information  processing  (IP) ,  or 
"lessons  learned”  (LL)  "customer”  organizations,  concerning 
the  agency's  IP  (or  LL)  efforts? 

(p)  Feedback  from  "Staff"  Customers  -  does  the  agency 
solicit  (or  obtain)  "IP”  (or  "LL")  feedback  or  comments  from 
customer  agency  "support”  and  "executive”  personnel? 

The  numerical  values  used  to  show  whether  or  not  the 
agency  "possessed"  (or  "performed")  the  characteristics  (or 
actions)  called  for  by  the  criteria  were  1  -  to  indicate 
"Yes,"  and  0  -  to  indicate  "No."  The  "numbers"  for  each 
agency  (using  the  specific  criteria  as  the  "independent 
variable")  are  shown  in  Table  l.  The  agencies  are  shown  in 
the  order  they  were  profiled  in  the  Findings  section;  the 
criteria  are  shown  in  the  order  they  were  listed  in  the 
previous  paragraph. 
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TABLE  1 


RESULTS  FROM  NOMINAL  SORTING 
(FOR  FIRST  EIGHT  CRITERIA) 

AGENCY  ALD  TQ  GAO  IGY  XRM  CY  TS  TA 
CRITERIA 

PM  Access  1  11  1  1  111 

Obtained  1  11  1  1  111 

b  d 

Struct  LL  11011100 

d 

Formal  LL  100  01  000 

a  c 

Modified  .1  11  1  1  011 

a  c 

Maintained  1  11  1  1  oil 

a 

Distributed  1  11  1  1  111 

a  c 

Decis  Makers  1  11  1  1  100 

a  Applies  specifically  to  TQ's  current  "Wins"  efforts 
b  Applies  specifically  to  IGY's  TIG  Brief  articles 
c  Applies  specifically  to  CY's  PM  Seminars 
d  does  not  include  Tacit  Rainbow  lessons  learned  efforts 
e  applies  because  of  GAO  report  distribution  problems 
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TABLE  1  (CONTINUED) 


RESULTS  FROM  NOMINAL  SORTING 
(FOR  SECOND  EIGHT  CRITERIA) 


AGENCY 

ALD 

TQ 

GAO 

IGY 

XRM 

CY 

TS 

TA 

CRITERIA 

a 

e 

b 

c 

Dec is  Implem 

1 

1 

0 

1 

1 

1 

0 

0 

a 

e 

b 

c 

Staff  Gets 

1 

1 

0 

1 

0 

0 

0 

0 

a 

c 

Metrics 

1 

0 

0 

0 

1 

0 

0 

0 

a 

c 

Quantity 

1 

0 

0 

0 

1 

0 

0 

0 

a 

c 

Quality 

1 

0 

0 

0 

1 

0 

0 

0 

a 

Form  Feedbk 

1 

0 

0 

•  0 

0 

0 

0 

0 

a 

Solic  Fdbk 

0 

0 

0 

0 

0 

0 

0 

0 

a 

Staff  Fdbk 

1 

0 

0 

0 

0 

0 

0 

0 

a  Applies 

specifically  to 

TQ'S 

current  "Wins"  i 

efforts 

b  Applies 

specifically  to 

IGY' 

s  TIG 

Brief 

articles 

c  Applies  specifically  to  CY ' s  PM  Seminars 
d  does  not  include  Tacit  Rainbow  lessons  learned  efforts 


e  applies  because  of  GAO  report  distribution  problems 


99 


The  results  shown  in  the  table  were  then  scored,  based 
on  a  subjective  percentage  figure  as  applied  to  two  sets  of 
sums  -  the  totals  per  agency,  and  the  totals  per  criteria. 

Assuming  that  the  "best”  possible  score  for  an  agency 
"total"  would  be  16  -  the  number  of  criteria  -  and  that  the 
"best  possible  score  for  a  criteria  "total"  would  be  8  -  the 
number  of  agencies  -  then  the  researcher  selected  80%  (of 
the  "best"  possible  scores)  as  being  the  minimum 
"acceptable"  percentage  needed  for  the  total  scores.  The 
rationale  for  the  "80%"  figure  is  based  partly  upon  well- 
understood  military  effectiveness  and  reliability  planning 
procedures;  many  military  weapons  development  specifications 
are  known  to  apply  percentages  of  95%  or  higher  as  the 
"minimum  required"  effectiveness  or  reliability.  However, 
this  investigation  is  mainly  considering  processes,  which 
are  generally  not  as  well  understood  in  terms  of  extremely 
accurate  "measuring"  criteria.  Therefore,  this  researcher 
added  a  "safety  factor"  of  15%  below  the  popular  95%  ,  to 
compensate  for  "process  uncertainties." 

The  results  for  the  agency  scoring  are  shown  in  Table 
2,  while  the  results  for  the  criteria  scoring  are  shown  in 
Table  3;  "Min  Accept"  is  the  minimum  acceptable  score,  80% 
of  16  (rounded  to  the  nearest  whole  number) ,  while  the 
letters  A  through  P  are  the  criteria  (  where  A  stands  for  PM 
Access,  B  stands  for  Obtained,  and  so  forth) . 
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TABLE  2 


RESULTS  OF  AGENCY  SCORING 


AGENCY 

SCORES 

Min  Accept 
Actual 


ALD 


TQ 


GAO 


IGY 


XRM 


CY 


TS 


TA 


13 

15 


13 

9 


13 

6 


13 

9 


13 

12 


13 

6 


13 

5 


13 

5 


TABLE  3 

RESULTS  OF  CRITERIA  SCORING 


(FIRST 

EIGHT 

CRITERIA) 

CRITERIA 

A 

B 

C 

D 

E 

F 

G 

H 

SCORES 

Min  Accept 

6 

6 

6 

6 

6 

6 

6 

6 

Actual 

8 

8 

4 

2 

7 

7 

6 

6 

TABLE  3  (CONTINUED) 

RESULTS  Of  criteria  SCORING 
(SECOND  EIGHT  CRITERIA) 


CRITERIA 

SCORES 

Min  Accept 
Actual 


I  J 


K  L  M  N 


0  P 


6  6  6  6 

5  3  2  2 


6  6  6  6 
2  10  1 


The  reader  may  now  be  asking  what  these  scores 
explain.  Basically,  these  scores  are  a  "first  level  warning 
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indicator,”  that  alert  the  researcher  (and  the  reader)  that 
many  agencies  do  not  appear  to  possess  most  of  the 
characteristics  that  the  criteria  would  indicate  are  needed 
for  a  "good”  lessons  learned  process.  At  the  same  time,  the 
lack  of  "use”  of  certain  criteria  could  indicate  problems 
with  the  "appropriateness”  of  that  criteria  in  the  "real 
world.”  Note  that  a  "visual”  inspection  of  Table  1  would 
also  have  "pointed  out”  the  various  "locations”  where  there 
were  many  "zeros”  and  few  "zeros;"  however,  the  computations 
used  for  Tables  2  and  3  are  less  subject  to  "visual  acuity" 
problems,  and  can  also  be  easily  verified  by  other  people. 

In  both  cases,  the  researcher  "re-reviewed"  the 
"qualitative”  information  obtained  in  the  investigation,  to 
develop  his  conclusions  and  recommendations  for  further 
actions.  Those  topics  are  discussed  in  the  following 
chapter. 

Summary 

This  chapter  presented  the  detailed  findings  about  the 
eight  sample  agencies,  in  particular  their  "information 
processing"  and  "lessons  learned"  operations.  It  explained 
the  nominal  sorting  techniques  used  by  the  researcher  to 
analyze  the  findings,  and  presented  the  results  of  the 
scoring  process  in  Tables  1  through  3.  Chapter  V  will 
provide  conclusions  derived  from  the  analysis, 
recommendations  to  improve  the  various  agencies'  "lessons 
learned"  processes,  and  suggestions  for  future  research. 
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V.  Conclusions  and  Recommendations 


Introduction 

This  chapter  shall  discuss  the  overall  conclusions 
reached  about  the  sample  agencies'  "lessons  learned" 
process.  It  shall  explain  and  recommend  changes  (to  the 
"lessons  learned"  processes)  that  have  the  potential  to 
improve  the  processes.  It  shall  also  provide 
recommendations  for  future  research  on  this  topic. 

Conclusions 

There  were  several  "overall"  conclusions  (about  the 
sample  agencies'  "lessons  learned"  efforts)  that  were 
readily  reached,  once  the  detailed  findings  were  reviewed 
and  the  analysis  performed.  Some  of  these  conclusions  are: 

(a)  All  of  the  sampled  agencies  regularly  obtain  (or 
can  obtain)  the  detailed  programmatic  information  needed  as 
"raw  material"  for  RD&A  project  management  "lessons 
learned;" 

(b)  All  of  the  agencies  sampled  have  (or  have  access 
to)  acquisition  personnel  who  have  the  knowledge  and 
experience  needed  to  identify  and  create  "lessons  learned" 
(from  the  obtained  programmatic  information  and  their  own 
backgrounds) ; 

(c)  Most  of  the  sampled  agencies  do  not  appear  to 
regard  providing  "lessons  learned"  (or  similar  information) 
for  agencies  outside  their  immediate  area  of  responsibility 
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as  a  high  priority  -  since  few  of  those  agencies  had  any 
sort  of  "lessons  learned"  effort,  formal  or  otherwise; 

(d)  Much  of  the  "information"  or  "lessons  learned" 
processing  that  many  of  the  sampled  agencies  do  perform  is 
oriented  for  the  "executive  level"  agencies  and  individuals; 
"middle"  and  "lower"  levels  of  management  (agencies  and 
individuals)  do  not  appear  directly  involved  and  therefore 
may  not  significantly/directly  benefit  from  such  processing; 

(e)  The  measurement  of  "information  transfer"  and/or 
"lessons  learned"  process  effectiveness  seems  to  be  a 
concern  only  for  those  sample  organizations  who  have  direct 
responsibility  for  such  processes  -  only  those  agencies  have 
developed  (or  are  developing)  such  "metrics." 

Recommended  Chances 

Any  attempts  to  improve  the  USAF's  "lessons  learned" 
processes  should  be  examined  from  the  "product"  and 
"process"  viewpoints  -  except  in  this  instance,  the  "lessons 
learned"  processes  are  themselves  the  "product,"  and  the  Air 
Force  "culture"  is  the  "process."  Essentially,  this  means 
that  the  "long-term"  approach  to  "lessons  learned"  issues 
will  require  changes  in  the  way  people  in  the  USAF  (and  all 
other  organizations  that  work  with  them)  view  their  roles 
and  responsibilities  (the  "culture") .  With  that  thought 
kept  in  mind,  this  researcher  has  recommended  several 
actions  that  would  change  the  USAF's  "ways  of  doing 
business;"  such  changes  would  have  the  potential  to  improve 
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"lessons  learned"  processes  as  they  relate  to  RD&A  project 
managers.  These  actions  are: 

(a)  Conduct  (a)  detailed  survey (s)  -  using 
questionnaires  or  semi-structured  interviews  -  of  all  RD&A 
project  management  personnel  in  certain  randomly  selected 
acquisition-related  organizations  (such  as  SPOs) ,  to  get 
feedback  directly  from  the  people  who  should  be  "customers" 
(and  sources)  for  "lessons  learned."  The  goal  of  such  (a) 
survey (s)  would  be  to  discover  the  ways  in  which  various 
levels  of  RD&A  managers  believe  they  gain  their  acquisition 
management  knowledge,  and  what  role  the  various  "lessons 
learned"  programs  play  in  the  "acquiring"  of  that  knowledge. 
To  achieve  this  goal  the  surveys  would  seek  to  determine: 
"what"  those  management  personnel  know  about  "lessons 
learned"  efforts;  to  what  degree  they  support  these 
processes;  what  faults,  problems,  concerns  (or  uses  and 
advantages)  they  see  in  the  current  "lessons  learned" 
programs;  and  also  whether  other,  "non-acquisition"  agencies 
such  as  the  IGs  or  GAO  are  considered  to  be  sources  of 
"useful"  management  information.  The  surveys  would 
certainly  ask  the  recipients  to  provide  comments  and 
suggestions  to  improve  "the  information  transfer"  and 
"lessons  learned"  processes. 

(b)  Make  "lessons  learned"  (LL)  an  integral  part  of 
the  "Total  Quality"  philosophy.  Significant  improvements 
have  been  made  in  various  acquisition  operations  once  the 
principles  and  support  structures  of  TQ  have  been  used  to 
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address  problems.  •'Learn  from  others  and  do  it  better'* 
would  be  a  potential  slogan  for  a  "combined"  LL/TQ 
philosophy;  a  more  tangible  step  would  be  to  form  a  Critical 
Process  Team  that  would  be  charged  with  developing 
"improvements  to  existing  knowledge  transfer  processes." 

Such  a  CPT  team  could  be  started  "locally"  but  would  have  to 
be  augmented  with  support  from  command  level  and  Systems 
Division  level  agencies. 

(c)  Expand  the  responsibilities  and  resources  of  the 
Program  Management  DCS  offices  (or  their  equivalent 
organizations)  to  encompass  "creating  and  providing"  program 
management  lessons  learned.  These  "functional"  offices 
would  be  allowed  to  "look  across"  the  various  RD&A  programs 
ongoing  at  their  Systems  Divisions,  then  use  that  "across" 
perspective  to  create  program  management  "lessons  learned." 
The  "perspective"  concept  is  analogous  to  the  "overview" 
idea  used  by  the  AFPEOs  -  except  that  the  PM  DCSes  would  not 
be  responsible  for  program  execution,  but  for  "learning 
from"  and  "providing  advice  to"  the  program  offices.  The 
knowledge  and  "lessons"  resulting  from  such  access  could  be 
provided  to  individual  agencies  through  the  existing 
"program  consultant"  role,  as  well  as  to  individual  project 
management  personnel  via  "home  office"  meetings,  messages, 
or  training  sessions. 

(d)  Institute  "lessons  learned"  usage  and  creation 
requirements  as  part  of  the  various  Program  Reviews  and 
Milestone  Decisions  that  most  RD&A  programs  have  to  satisfy. 
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If  most  RD&A  agencies  had  to  use  or  create  lessons  learned 
on  a  regular  basis,  then  support  for  "lessons  learned"  could 
become  "part  of  the  job,"  rather  than  an  "additional  duty." 

(e)  Bring  "evaluator"  agencies  "into"  the  acquisition 
process,  preferably  as  early  as  possible.  Many  of  these 
agencies'  "inspectors"  have  been  trained  as  acquisition 
professionals  -  for  example,  IGY's  SAMI  team  members  are  all 
chosen  from  the  ranks  of  "veteran"  acquisition  community 
members  (74)  -  and/or  have  vast  professional  experience  in 
dealing  with  acquisition  issues;  using  them  as  "graybeard" 
advisors  (particularly  early  in  the  development  cycle)  would 
tap  that  expertise,  and  perhaps  give  their  agencies  a 
greater  appreciation  for  the  problems  that  often  arise  in 
development  and  acquisition  programs.  As  a  part  of  such 
"integration,"  arrange  for  GAO  and  AFISC  "RD&A  related" 
reports  to  contain  "lessons  learned"  sections,  and  for  those 
reports  to  be  provided  to  "the  lowest  levels"  of  Air  Force 
RD&A  organizations  (not  just  HQs) ,  so  that  the  "widest 
dissemination  of  this  knowledge"  can  be  achieved. 

(f)  Have  acquisition  agencies  work  with  the  various 
training  agencies,  such  as  DSMC  and  AFIT,  to  develop 
additional  or  alternate  "mediums"  for  providing  "lessons 
learned."  For  example,  AFIT/LSY  has  "contracted  out"  for 
development  of  a  "computer-interactive"  training  tool,  to 
support  its  Systems  100-200  acquisition  management  courses. 

(g)  Expand  use  of  the  concept  of  "interactive 
seminars,"  to  where  various  acquisition  "scenarios"  or 
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issues  (that  would  "expose”  participants  to  "lessons 
learned")  are  presented  in  a  "simulated  acquisition 
experience."  This  is  basically  the  same  principle  used  by 
aircrews  when  they  train  using  their  aircraft  simulators 
(although  admittedly  seminars  aren't  quite  so  "hands-on" 
oriented)  -  mistakes  can  be  made,  "lessons"  can  be  learned, 
without  the  impacts  (financial  and  otherwise)  that  "real 
life"  errors  would  cause. 

Most  of  these  changes/actions  just  described  would 
require  significant  time  (and  in  certain  cases  resources)  to 
implement.  In  the  "immediate"  time  frame,  several  actions 
could  be  taken  to  improve  the  versatility  and  potential 
effectiveness  of  existing  (or  developing)  "lessons  learned" 
programs.  Some  of  the  possible  actions  include: 

(a)  Expand  the  AFSC  Program  Managers  Lessons  Learned 
Process,  to  allow  lessons  that  were  created  by  people  other 
than  just  program  directors.  The  current  arrangement  does 
not  consider  the  extremely  time-consuming  duties  most 
program  directors  face  -  thus,  many  PDs  may  feel  they  should 
"fight  the  fires  first"  and  provide  "lessons"  later.  If  so, 
then  few  lessons  would  get  submitted  (in  point  of  fact,  only 
"a  few"  lessons  have  been  provided  to  date) .  However,  it  is 
often  much  more  feasible  for  PDs  to  "revise  and  alter"  draft 
documents,  rather  than  creating  the  documents  themselves. 

If  this  avenue  were  allowed  for  the  PMLLP,  more  lessons 
might  be  submitted  -  particularly  if  PDs  tasked  their 
managers  to  provide  draft  LLs  at  regular  intervals. 
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(b)  Allow  "background  papers"  and  longer  documents  to 
be  input  into  the  Air  Force  Lessons  Learned  Program  (AFLLP) 
database.  The  current  format  for  AFLLP  lessons  is  suitable, 
when  viewed  in  the  role  of  a  "synopsis"  or  an  abstract. 
However,  many  individuals  (inside  and  outside  the  sampled 
agencies)  have  expressed  concerns  that  the  complexities  and 
"nuances"  of  most  RD&A  efforts  are  extremely  difficult  to 
"condense"  into  the  one  or  two  pages  that  are  currently  the 
limit  for  a  "lesson  learned."  Therefore,  certain  details 
may  be  "left  out"  or  "overlooked"  that  could  impact  the 
usefulness  of  the  "lesson."  An  example  might  be  the  lesson: 
"contractor-government  working  groups  should  be  used  to 
coordinate  development  on  full-scale  development  programs." 
If  the  lesson  was  written  based  upon  a  "cost-plus"  type  of 
development,  the  lesson  might  not  explain  the  type  of 
contract  used  and  the  contractual  ramifications  a  person  on 
a  "fixed-price"  contract  might  encounter  if  they  attempted 
to  use  this  lesson  after  contract  award.  A  "background 
paper"  or  longer  input,  concerning  such  a  lesson,  would  be 
able  to  include  much  of  the  "context"  information  that  would 
explain  the  contract  background,  and  the  impacts  of  such  an 
action  on  the  contract.  In  addition,  allowing  longer 
documents  (such  as  the  Tacit  Rainbow  Mission  Planning  System 
"lessons  learned"  document)  would  provide  a  larger,  more 
"contextual"  oriented  information  source.  In  those  cases 
where  the  document  is  too  long,  establish  a  physical 
location  (such  as  a  technical  library  or  possibly  a  history 
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office)  to  hold  the  "paper"  copies.  "Off-site"  information 
could  be  referenced  in  the  "normal"  lessons  learned  format. 

(c)  Restart  the  Project  Officer's  Group  (POG) 
meetings.  These  POG  sessions,  attended  by  AFPEO  and  AFSC 
"action  officers,"  would  be  an  excellent  forum  for 
discussing  common  issues  and  concerns,  and  could  be  used  by 
the  PEOs  to  provide  "lessons  learned"  through  their  action 
officers;  the  AFSC  "recipients"  of  those  lessons  could  in 
turn  (as  appropriate)  provide  the  "lessons  learned"  through 
the  PMLLP  to  other  acquisition  organizations.  In  addition, 
such  meetings  would  maintain/enhance  the  AFPEO/HQ  AFSC 
communications  that  are  currently  "decreasing,"  because  of 
the  HQ  AFSC  "move"  to  Wright-Patterson  AFB. 

The  previously  described  changes  are  but  a  few  of  many 
possible  changes.  This  researcher  feels  these  changes  are 
"doable"  and  should  improve  the  USAF's  "lessons  learned" 
concepts  and  processes. 

Recommended  Future  Research 

This  investigation  has  essentially  been  the  first  phase 
of  a  multi-phase  effort  to  examine  existing  and  potential 
"lessons  learned"  processes  and  support  improvements  to 
those  processes.  Time  and  resource  constraints  prevented  a 
full  investigation  of  "all"  possible  "lessons  learned" 
processes  and  also  the  initiation  of  the  second  phase  of  the 
effort  -  "customer"  surveys.  In  this  second  "part,"  a 
survey  such  as  the  one  described  in  the  Recommended  Changes 
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subsection  would  be  designed,  developed,  and  conducted, 
using  a  representative  sample  of  RD&A  agencies  as 
"recipients.”  The  results  would  be  analyzed,  to  form  the 
basis  for  additional  recommended  changes.  Such  a  survey 
could  then  be  modified  and  redone  to  gather  information 
about  the  results  of  any  changes  implemented  after  the 
initial  survey  (such  as  changes  recommended  in  this  study, 
or  changes  started  as  "resulting  by-products"  of  this 
investigation) . 
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APPENDIX  A 


TOPIC  CHECKLIST 


TOPIC  #1:  Possibility  For  RD&A  "Lessons  Learned" 

Purpose  of  Interview  Questions;  To  identify  the  purpose  and 
general  features  of  the  specific  agency  being  investigated; 
the  answers  should  indicate  whether  or  not  the  agency  is  (or 
could  be)  a  "processor"  of  RD&A  program  information  that  is 
(or  should  be)  used  for  project  management  "lessons 
learned."  Such  data  would  also  indicate  if  that  agency  had 
been  "placed"  in  the  appropriate  sampling  subdivision. 

Objective  A:  Identify  the  purpose/mission  of  the  sample 
agency  being  investigated. 

-  Oversight/Management? 

-  Provider  of  Resources/Support? 

-  Evaluator/Advisor? 

-  'Gathering  of  Information? 

-  Other? 

Objective  B;  Determine  the  focus  of  sample  agency's 
interfaces  with  USAF  organizations  (who  they  work  with  to 
accomplish  their  mission) . 

-  RD&A  organizations? 

-  Major  Commands  (MAJCOMMs)? 

-  Weapon  systems  "User"  or  "Maintainer"  organizations? 
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others? 


Objective  C:  Identify  the  justifications  used  by  the  sample 
agency  to  obtain  program  information  from  RD&A  agencies. 

-  Common  goals/purposes? 

-  "Corporate”  Structure? 

-  Laws  or  regulations? 

-  Policies  or  agreements? 

-  Personal  relationships  or  "expected"  behavior? 

-  Others? 

Objective  D;  Find  out  the  relevant  "subject  areas"  the 
sample  agency  is  interested  in  when  it  interfaces  with  USAF 
RD&A  organizations. 

-  Financial  operations? 

-  Project  management? 

-  Technology  development? 

-  Logistics? 

-  Others? 

Objective  E:  Identify  whether  or  not  sample  agency  can  (or 
does)  provide  RD&A  information  it  obtains  to  agencies 
outside  its  "hierarchy  of  command"  (which  is  those  agencies 
directly  "above"  and  "below"  the  subject  agency  in  its  own 
organizational  structure) 

-  Agencies  with  similar  or  complementary  missions? 

-  Organizations  with  missions  similar  to  those  of 

"interfaced"  USAF  RD&A  agencies? 

-  Others? 
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Objective  F;  Determine  whether  or  not  the  sample  agency  has 
a  specifically  identified  "lessons  learned”  (or  equivalent 
term)  process. 

-  Formal  "Lessons  Learned"  (or  equivalent  term)  Program? 

-  Official,  documented  policies,  procedures,  and 

techniques  for  acquiring,  maintaining,  and 
disseminating  "lessons  learned"  (or  equivalent 
term)  information? 

-  Informal  "lessons  learned”  (or  equivalent  term) 

efforts? 

-  Unofficial  (or  undocumented)  policies,  agreements, 

practices,  and  methods  for  obtaining,  formatting, 
and  "passing  on"  information  such  as  "things  to 
avoid,"  "lessons  learned,"  "best  ways  of 
accomplishing  (Task  X),"  and  similar  knowledge? 
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TOPIC  #2;  Information  Processing 


Purpose  of  Interview  Questions;  To  determine:  the  specific 
types  of  organizations  (and  individuals)  from  which  the 
sample  agency  obtains  RD&A  program  information;  the  methods 
that  agency  uses  to  ''modify"  and  "comment  on"  the 
information  it  obtains;  the  ways  in  which  the  agency  "passes 
on"  the  "altered"  information  to  other  org^.nizations.  The 
answers  should  describe  the  agency's  methods  of  processing 
(acquiring,  maintaining,  and  disseminating)  actual  (or 
potential)  "lessons  learned"  information. 

Objective  A;  Identify  the  sample  agency's  sources  of  RD&A 
program  information. 

-  Program  Directors/Managers? 

-  Program  Personnel? 

-  On-Site  Representatives? 

-  Different  Non-Program  Agencies? 

-  Different  Program  Offices/Organizations? 

-  Others? 

Objective  B;  Describe  the  methods  used  by  the  sample  agency 
to  identify  and  communicate  its  information  "needs"  to  the 
information  sources. 

-  Written  requests? 

-  Telephone? 

-  Videoteleconference? 
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Personal  contact? 


-  Other? 

Objective  C;  Determine  the  categories  of  the  "subjects  of 
interest"  information  requests  made  by  the  sample  agency  (to 
the  RD&A  organization) . 

-  Program/project/subproject? 

-  Command/ Systems  Division/SPO  (or  equivalent 

hierarchy) ? 

-  Functional  specialty  (engineering,  management, 

others)? 

-  Special  Interest  (development  testing  and  evaluation, 

information  management,  maintenance,  others)? 

-  Others? 


Objective  D;  Describe  the  "extent  format"  the  RD&A 

information  (that  the  sample  agency  obtains)  is  in. 

-  "Executive  Summary"  (primarily  main  points/issues)? 

-  "Background/Status  Report"  (main  points  plus  some 

"details"  and  general  background) ? 

-  "Worker-Level  Review"  (main  points  plus  extensive 

"details"  and  background)? 

-  Other? 


Objective  E:  Identify  the  medium (s)  in  which  RD&A  program 
information  is  furnished  to  the  sample  agency. 

-  Written  documents? 

-  Verbal  (telephone)? 

-  Personal  contact  (meetings,  briefings,  others)? 

-  Audiovisual  media? 

-  Electronic  transmission? 
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-  Magnetic  media? 

-  Other? 

Objective  F;  Determine  the  types  of  changes  that  the  sample 
agency  makes  (or  has  the  furnishing  agency  make)  to  the 
"regular"  content/ structure  of  furnished  RD&A  information. 

-  Reorder/reformat? 

-  Rephrase/rewrite? 

-  Edit  (remove  information)? 

-  Add  additional  comments/data? 

-  Emphasize  important  ideas/ features? 

-  Key  Points  (or  "lessons  learned")? 

-  No  changes? 

-  Other? 

Objective  G;  Describe  the  medium(s)  in  which  the  sample 
agency  maintains  the  (changed)  information. 

-  "Paper"  copies? 

-  Electronic  database/f iles/software? 

-  Magnetic  or  optical  discs? 

-  Audiovisual  media? 

-  Personal  memories? 

-  Other? 

Objective  H:  Identify  the  "materials"  (or  document  types) 
that  the  sample  agency  keeps  this  information  in. 

-  Reports,  briefings,  background  papers,  others? 

-  Logs,  working  notes,  drawings,  diagrams,  others? 
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-  Official  Forms,  files,  notebooks,  books,  others? 

-  Other? 

Objective  I:  Determine  the  subject  "indexes"  used  by  the 
sample  agency  to  store/maintain  the  (changed)  information. 
-  Program/project/ subproject? 

-  Service/Command/Systems  Division/SPO  (or  equivalent 

organizational  hierarchy)? 

-  Functional  category  (financial,  management,  others)? 

-  Special  Interest  (specific  technology,  support 

equipment,  others)? 

-  Alphabetical  order/program  element/other  codes? 

-  Other? 

Objective  J :  Determine  the  "extent  format"  used  by  the 

sample  agency  to  store/maintain  this  information. 

-  "Executive  Summary"  (primarily  main  points/issues)? 

-  "Background/Status  Report"  (main  points  plus  some 

"details"  and  general  background) ? 

-  "Worker-Level  Review"  (main  points  plus  extensive 

"details"  and  background)? 

-  Other? 

Objective  K;  Describe  the  methods  used  by  the  sample  agency 
to  find  and  retrieve  stored  RD&A  (changed)  information  when 
it  is  needed. 

-  Electronic  "key  word"  search  software? 

-  Personal  research  in  files? 

-  Card  catalog/file  plan? 

-  Other? 
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Objective  L;  Determine  the  categories  in  which  the 

(changed)  RD&A  information  is  disseminated  by  the  sample 

agency  (to  other  organizations) . 

-  Program/project/ subproject? 

-  Command/ Systems  Division/SPO  (or  equivalent 

hierarchy) ? 

-  Functional  specialty  (engineering,  management, 

others) ? 

-  Special  Interest  (development  testing  and  evaluation, 

information  management,  maintenance,  others)? 

-  Others? 

Objective  M:  Describe  the  medium (s)  in  which  the  sample 
agency  "sends  out"  the  (changed)  information. 

-  "Paper"  copies? 

-  Electronic  database/ files/software? 

-  Magnetic  or  optical  discs? 

-  Audiovisual  media? 

-  Telephone  (verbal)? 

-  Personal  contact? 

-  Other? 

Objective  N:  Identify  the  "materials"  (or  document  types) 

that  the  sample  agency  provides  this  information  in. 

-  Reports,  briefings,  background  papers,  others? 

-  Pamphlets,  brochures,  periodicals,  "news  updates", 

presentations,  others? 

-  Official  Forms,  files,  notebooks,  books,  others? 

-  Other? 
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Objective  O;  Identify  the  methods  used  by  the  sample  agency 
to  decide  what  organizations/ individuals  should  receive  (or 
have  access  to)  the  (changed)  information. 

-  Distribution  List? 

-  Written  requests/direction? 

-  Verbal  requests/direction? 

-  Access  codes? 

-  Mission  requirements? 

-  Other? 

Objective  P:  Determine  what  types  of  agencies/ individuals 
receive  the  (changed)  RD&A  information  that  this  sample 
agency  disseminates. 

-  all  U.  S.  Government  personnel? 

-  all  Government  contractors? 

-  Congress/the  President/Secretary  of  Defense/ others? 

-  Program  Directors/Managers/Executive  Officers/others? 

-  Other  "information  processing"  agencies? 

-  Most  MajComms/SPOs/Labs/ other  RD&A  personnel? 

-  Anyone  who  asks  (or  Other)? 
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TOPIC  #  3:  Process  Evaluation  Criteria 


Purpose  of  Interview  Questions:  To  describe  the  various 
criteria  the  sample  agency  uses  to  judge  the  effectiveness 
of  its  "information  processing”  efforts.  The  answers  should 
provide  an  indication  of  whether  or  not  the  sample  agency 
should  adopt  additional  criteria/metrics  (and  if  a  draft  set 
of  such  metrics  -  such  as  a  customer  survey?  -  should  be 
considered/drafted/developed  as  part  of  this  study) . 

Objective  A:  Identify  the  various  types  of  "quantitative 
measurement”  criteria  used  to  evaluate  the  sample  agency's 
"information  processing"  efforts. 

-  During  a  Given  Time  Period, 

-  For  "All"  (an  aggregate  total)  Types  of  (Changed) 

RD&A  Information: 

-  #  of  agencies  that  provide? 

-  #  of  organizations  that  receive? 

-  #  of  agencies  that  request? 

-  #  of  individual  submissions? 

-  #  of  individual  "receptions"? 

-  #  of  individual  requests  (for)? 

-  #  of  comments/ feedback  from  agencies  and 

organizations? 

-  #  from  providers? 

-  #  from  receivers? 

-  #  from  requesters? 

-  #  of  comments/ feedback  from  individuals? 

-  #  from  providers? 
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-  #  from  receivers? 

-  #  from  requesters? 

-  For  a  Certain  Category  (management,  logistics, 

financial,  others)  of  (Changed)  RD&A  Information: 

-  #  of  agencies  that  provide? 

-  #  of  organizations  that  receive  ? 

-  #  of  agencies  that  request? 

-  #  of  individual  submissions? 

-  #  of  individual  "receptions"? 

-  #  of  individual  requests  (for)? 

-  #  of  comments/ feedback  from  agencies  and 

organizations? 

-  #  from  providers? 

-  #  from  receivers? 

-  #  from  requesters? 

-  #  of  comments/ feedback  from  individuals? 

-  #  from  providers? 

-  #  from  receivers? 

-  #  from  requesters? 

-  Other? 

Objective  B:  Determine  the  various  types  of  "qualitative 

measurement"  criteria  used  to  evaluate  the  sample  agency's 

"information  processing"  efforts. 

-  Identity  of  agencies/individuals  that  provide  the 

(changed)  RD&A  information? 

-  Identity  of  organizations/individuals  that  receive 

(changed)  RD&A  information? 

-  Identity  of  agencies/individuals  that  request  (changed) 

RD&A  information? 
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-  Are  these  agencies/ individual  "inside"  or  "outside"  the 

sample  agency's  "hierarchy?" 

-  Content  of  comments/ feedback  from  "customers"  (the 

"changed"  information  providers,  receivers,  requesters, 
and  others)? 

-  "Procedural"  (process  earlier/ later,  more  copies 

for  one  agency,  others)? 

-  "Substance"  (add  more  customers,  improve  clarity, 

get  information  from  more  or  different  sources, 
focus  on  different  subjects,  and  so  forth)? 

-  Fiscal  Constraints  (is  there  money  for  new 

software,  hardware,  travel,  and  such)? 

-  Identity  of  agencies  that  give  comments/ feedback? 

-  Provider? 

-  Receiver? 

-  Requester? 

-  Others? 

- Inside  or-  Outside  Hierarchy? 

Objective  C;  Describe  the  techniques  used  by  the  sample 

agency  to  obtain  comments/ feedback  about  their  "information 

process"  from  their  "customers." 

-  Unsolicited  Comments/ Feedback 

-  Questions,  concerns,  inquiries,  suggestions, 

direction,  other? 


-  Solicited  Comments/Feedback 

-  Informal  ("what  are  you  doing  with  the  information, 

is  information  useful,  do  you  have  any  thoughts  or 
comments  on  our  methods")? 

-  Verbal,  personal  contact,  request  memo,  other? 

-  Formal  (for  the  record) 

-  Customer  comment  forms? 
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-  Questionnaires? 

-  Customer-generated  "usefulness”  reports? 

-  Structured  interviews? 

-  Meetings  with  customers  (with  meeting  minutes, 

memos  for  record,  other)? 

-  Customer-generated  "progress”  briefings? 

-  No  formal  methods? 

-  Other? 

Other? 
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TOPIC  #  4 ;  Changes/ Improvements  to  Information  Processing 


Purpose  of  Interview  Questions;  To  profile  current  and 
proposed  changes/ improvements  to  the  sample  agency's 
information  processing  practices.  This  information  will  be 
used  as  part  of  the  basis  for  the  researcher's 
recommendations  regarding  for  changes/ improvements  to  the 
sample  agency's  information  processing  and  "lessons  learned" 
practices. 

Objective  A:  Identify  recent  (within  the  previous  6-9 
months)  changes  and  improvements  that  have  been  suggested  by 
agencies/individuals  that  are  not  part  of  .ne  sample  agency. 

-  Who  made  suggestions? 

-  Providers-,  receivers,  requesters,  others? 

-  What  types  of  Suggestions? 

-  "Procedural"  (process  earlier/later,  more  copies 

for  one  agency,  others)? 

-  "Substance"  (add  more  customers,  improve 

clarity,  get  information  from  more  or  different 
sources,  focus  on  different  subjects,  and  so 
forth) ? 


Objective  B;  Describe  the -changes  and  improvements  that  the 
sample  agency  is  implementing/considering. 

-  What  types  of  Suggestions? 

-  "Procedural"  (process  earlier/later,  more  copies 

for  one  agency,  others)? 

-  "Substance"  (add  more  customers,  improve 

clarity,  get  information  from  more  or  different 
sources,  focus  on  different  subjects,  and  so 
forth) ? 
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-  Who  suggested  these  changes/ improvements? 

-  Internal  (offices  or  individuals  within  the 

sample  agency? 

-  External  (Customer  agencies/individuals)? 

-  Other? 

Objective  C:  Determine  the  justifications  cited  by  the 
sample  agency  for  implementing/considering  the  changes  and 
improvements . 

-  Deficiencies  in  current  methods? 

-  New  capabilities  not  previously  available? 

-  Hardware? 

-  Software? 

-  Organizational  structure? 

-  Different  relationships  between  agencies  or 

individuals? 

-  Other? 

-  Direction  from  higher  levels  in  hierarchy? 

-  More  efficient  methods? 

-  Customer  "preference"? 

-  Other? 
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Appendix  B 

Excerpts  From  Selected  Sources 


1.  Air  Force  Form  1251,  Potential  Lessons  Learned  Submittal 
Record,  Jan  90  (page  128) 


2.  AFSCR  550-19,  Commander's  Policies  -  Program  Management 
Lessons  Learned  Process,  28  June  1991  (pages  129-130) 


3.  Air  Force  Acquisition  System  (Diagram)  (page  131) 
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RZCOM^ENDED  ACTION: 
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DEPARTMENT  OF  THE  AIR  FORCE 
Headquanos  Air  Force  Systeass  Ommaod 
Andrews  Air  Force  Base  DC  20334-5000 

Commander’s  Policies 


PROGRAM  MANAGEMENT  LESSONS  LEARNED  PROCESS 

1.  Megring  the  challenges  of  today’s  acquisition  environment  requires  us  to  avoid  past  mis¬ 
takes— our  Lessons  Learned.  To  facilitate  this  effort,  we’ve  pulled  together  xnsny  ideas  into 
an  integrated,  comprehensive  Ptogram  Management  Lessons  Learned  Process  (Atch  1). 

2.  Access  to  the  Air  Force  Lessons  Learned  database  has  been  simplified,  and  a  Program 
Director’s  (PD)  area  has  been  adderi  The  Program  Director’s  area  consists  of  unedited  Lessons 
Learned  disseminated  from  an  “as  it  happens/as  required”  elecnonic  information  network  (IN- 
FWET).  The  HQ  AFSC/XR  Lcssots  Learned  INPC^JET  contains  ctecuhve  level,  fast-breaking 
insights/esperiences  presided  by  the  Service  Acquisition  Etecuiives,  Program  Bcecuiive  Of¬ 
ficers,  axxl  the  Program  Directors.  Program  Directors  should  also  capture  aixl  share  with  their 
peers  the  Lessons  Learned  from  key  program  events.  The  media  used  could  be  videonqjes, 
inputs  to  HQ  AFSCTXR’s  Lessons  Learned  INFWET,  or  letters  to  mt  Ws  will  arrange  to 
share  them  throughout  the  command. 

3.  The  produa  divisions  will  est^lish  face-to-face  meetings  between  program  offices  for  dis¬ 
seminating  Lessons  Learned  informarina  One  form  of  these  meetings  is  PD  Qrcles.  a  forum 
for  discussing  cross-program  impacts  of  your  ongoing  activities  with  a  common  contractor  or 
subcontractor,  Discussions  between  the  product  divisions  or  within  a  product  division  result 
in  a  more  uniform  ccrpotaie  strategy  for  rivaling  with  a  common  contractor. 

4. 1  ctpea  Program  Direcasis  to  eacour^e  dieir  subordinates  to  support  the  Air  Force  Lessons 
( Program.  In  doing  so,  I  urge  you  to  use  appropriate  caution  when  dealing  with  sensitive' 
or  propriesny  data. 


5.  No  process  is  con^cte  without  metrics.  KQ  AFSC  will  repon  progress  on  the  segments 
we  are  responsfoie  for,  and  I  etpect  each  product  division  to  develop  metrics  and  to  report 
tbelr  pregresL  HQ  AF50XR  will  woi^  with  you  to  establish  standard  mesxics  and  will  revitalize 
this  critical  Lessors  Learned  process. 


I  »-  *  — s.  —  - - 

RONALD  W.  'iATES,  General,  USAF 
Commander 


1  Atch 

AFSC  Ptogiam  Managcmeffi 
Igsons  Learned  Prcc^ 


\FSCR  5S0-l» 
23  Juo*  1991 
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AFSCR  550-19  28  June  1991 


AFSC  PROGRAM  MANAGEMENTT  LESSONS  LEARNED  PROCESS 


OiBDbubaK  P  X: 

ABXXV,  AobM  AfB  TN  3709-5000  . 10 

ATOTXXS,  Ellia  AfB  PL  3Z54MOOO . 10 

AFFTDCS.  APB  CA  93S23-5000 . 10 

ASIVCS.  Wi!||«.Piomo«  APB  OH  4S43MS<B  . 10 

eSIVCL  lltf  AfB  MA  01731-5000 . 10 

FTOCS,  Wng|»P«Ma«  AfB  OH  454X54500 . 10 

HSDrCS,  BiwOi  APB  TX  7003-5000 . 10 

SSDCS.  La  Aaocia  APB  CA  90009-»M . 10 

HQ  AHSOIMP.  Nom  APB  CA  9240».X)0I  . 1 

AIX/ISE,  MawdI  APB  AL  36112-5564  . 1 


AFOC— AaOmn  APB  DC  1991 
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and  later  acted  as  program  manager. for  all  Air  Force 
development  efforts  in  the  Joint  Service  Small  Arms  Program. 
In  August  1986,  he  was  re-assigned  to  Aeronautical  Systems 
Division,  Wright-Patterson  AFB,  Ohio.  He  initially  served 
in  the  Strategic  Systems  Program  Office,  as  a  munitions 
subsystems  development  manager.  In  1988  he  became  a  member 
of  the  Special  Projects  Office.  There  he  managed  subsystems 
development,  and  later  contracts  changes,  for  various  highly 
advanced,  classified  aeronautical  systems  technologies, 
until  he  entered  the  School  of  Systems  and  Logistics,  Air 
Force  Institute  of  Technology,  in  May  1990. 
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